This  special  version  of  the  report  has  been  revised  to  omit  predecisional  data. 


Department  of  Defense 

distribution  statement  a 

Approved  for  Public  Release 
dtic  quality  inspected  a  Distribution  Unlimited 


ASnoo-oi-  nM3 


0417  084 


Additional  Copies 

Copies  of  the  report  can  be  obtained  from  the  Secondary  Reports  Distribution  Unit, 
Audit  Planning  and  Technical  Support  Directorate,  (703)  614-6303  (DSN  224-6303). 


Acronyms 

AFOTEC 

Air  Force  Operational  Test  and  Evaluation  Center 

ASPJ 

Airboume  Self-Protection  Jammer 

CDR 

Critical  Design  Review 

EMD 

Engineering  and  Manufacturing  Development 

GAO 

General  Accounting  Office 

IOT&E 

Initial  Operational  Test  and  Evaluation 

JTIDS 

Joint  Tactical  Information  Distribution  System 

LRIP 

Low-Rate  Initial  Production 

MILSTAR 

Military  Strategic  and  Tactical  Relay  Satellite 

OPTEVFOR 

Operational  Test  and  Evaluation  Force 

OSD 

Office  of  the  Secretary  of  Defense 

PLS 

Palletized  Load  System 

PRR 

Production  Readiness  Review 

SFW 

Sensor  Fuzed  Weapon 

TRR 

Test  Readiness  Review 

USD(A) 

Under  Secretary  of  Defense  for  Acquisition 

INSPECTOR  GENERAL 

DEPARTMENT  OF  DEFENSE 
400  ARMY  NAVY  DRIVE 
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November  9,  1993 

MEMORANDUM  FOR  UNDER  SECRETARY  OF  DEFENSE  FOR  ACQUISITION 

SUBJECT:  Audit  Report  on  Low-Rate  Initial  Production  in  Major  Defense  Acquisition 
Programs  (Report  No.  94-014) 


We  are  providing  this  final  report  for  your  information  and  use.  This  report 
addresses  the  effectiveness  of  DoD's  use  of  Low-Rate  Initial  Production  in  major 
Defense  acquisition  programs.  Management  comments  were  considered  in  preparing 
the  final  report  and  are  in  Part  IV,  Management  Comments. 

DoD  Directive  7650.3  requires  that  all  recommendations  be  resolved  promptly. 
Therefore,  we  request  that  the  Under  Secretary  of  Defense  for  Acquisition  provide 
comments  on  the  unresolved  recommendations  by  January  10,  1994.  If  you  concur, 
describe  the  corrective  actions  taken  or  planned,  the  completion  dates  for  actions 
already  taken,  and  the  estimated  dates  for  completion  of  planned  actions.  If  you 
nonconcur,  state  your  specific  reasons  for  each  nonconcurrence.  If  appropriate,  you 
may  propose  alternative  methods  for  achieving  desired  improvements.  The 
recommendations  are  subject  to  resolution  in  accordance  with  DoD  Directive  7650.3  in 
the  event  of  nonconcurrence  or  failure  to  comment.  We  also  ask  that  your  comments 
include  concurrence  or  nonconcurrence  with  the  material  internal  control  weaknesses 
highlighted  in  Part  I.  This  report  identifies  no  quantifiable  monetary  benefits. 

We  appreciate  the  courtesies  extended  to  our  audit  staff.  If  you  have  any 
questions  on  this  report,  please  contact  Mr.  Russell  A.  Rau,  Program  Director,  at 
(703)  693-0186  (DSN  223-0186)  or  Mr.  Harold  C.  James,  Project  Manager,  at 
(703)  614-3974.  Copies  of  the  final  report  will  be  distributed  to  the  organizations 
listed  in  Appendix  K. 

Robert  J.  Lieberman 
Assistant  Inspector  General 
for  Auditing 


This  special  version  of  the  report  has  been  revised  to  omit  predecisional  data. 


Office  of  the  Inspector  General,  DoD 


Report  No.  94-014 
(Project  No.  2AE-0026) 


November  9,  1993 


LOW-RATE  INITIAL  PRODUCTION  IN  MAJOR  DEFENSE 
ACQUISITION  PROGRAMS 


EXECUTIVE  SUMMARY 


Introduction.  Low-Rate  Initial  Production  (LRIP)  is  defined  as  the  production  of  a 
system  in  limited  quantity  to  provide  articles  for  operational  test  and  evaluation,  to 
establish  an  initial  production  base,  and  to  permit  an  orderly  increase  in  the  production 
rate  sufficient  to  lead  to  full-rate  production  upon  completion  of  operational  testing. 
Planning  for  LRIP  begins  early  in  the  acquisition  process.  Proposed  LRIP  quantities 
are  determined  during  Phase  I,  Demonstration  and  Validation.  Milestone  decision 
authorities  then  set  the  LRIP  quantity  at  the  Milestone  II,  Development  Approval.  The 
design,  testing,  and  production  preparation  efforts  necessary  to  support  entry  into  LRIP 
are  presently  part  of  Phase  II,  Engineering  and  Manufacturing  Development. 

Objectives.  The  primary  objective  of  the  audit  was  to  evaluate  the  effectiveness  of 
DoD's  use  of  the  LRIP  in  major  Defense  acquisition  programs.  We  also  evaluated 
compliance  with  the  internal  controls  associated  with  the  LRIP  process. 

Audit  Results.  LRIP  was  not  being  used  effectively  to  manage  program  risks  inherent 
in  both  the  entry  into  LRIP  and  the  transition  from  LRIP  to  foil-rate  production. 

o  All  seven  of  the  major  Defense  acquisition  programs  reviewed  entered  LRIP 
without  completing  at  least  some  prerequisites  in  design,  testing,  and  preparation  for 
production.  Premature  entry  into  LRIP  was  caused  by  inadequacies  in  the  milestone 
review  process,  regulations,  and  policy  guidance  for  LRIP .  As  a  result,  DoD  incurred 
unwarranted  program  risk  due  to  excessive  program  concurrency  (Finding  A). 

o  LRIP  acquisition  strategies  did  not  effectively  limit  production  quantities 
before  Milestone  III,  Production  Approval.  As  a  result,  the  Government  incurred  the 
excessive  program  risk  of  over-commitment  to  production  of  systems  that  have  not 
proved  their  technical  or  operational  suitability  or  production  readiness  and  that  may 
have  deficiencies  that  are  costly  and  difficult  to  fix  on  delivered  and  accepted  units 
(Finding  B). 

Internal  Controls.  The  audit  identified  material  internal  control  weaknesses. 
Finding  A  shows  that  Department  of  Defense  directives,  instructions,  and  standards 
provided  for  only  limited  guidance  and  oversight  regarding  transition  from  development 
to  LRIP.  Finding  B  identified  that  increased  Office  of  the  Secretary  of  Defense  and 
Military  Department  oversight  was  needed  to  ensure  that  LRIP  quantities  were  limited 
to  the  minimum  necessary  for  testing  and  production  base  purposes.  This  oversight 
will  ensure  that  all  prerequisites  are  met  before  making  the  low-rate  production 
decision.  These  internal  control  weaknesses  are  discussed  in  Part  I  of  this  report. 

Potential  Benefits  of  the  Audit.  Implementing  the  recommendations  in  this  report 
will  ensure  that  decisions  to  enter  and  continue  LRIP  production  are  made  when  folly 
supported  by  risk  assessments  based  upon  testing,  design,  and  production 
accomplishments.  Further,  LRIP  quantities  will  be  limited  to  the  minimum  necessary 
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to  provide  production  units  for  operational  test  and  evaluation,  to  establish  a  production 
base,  and  to  transition  to  full-rate  production.  Monetary  benefits  to  be  realized  by 
implementing  the  recommendations  were  not  readily  quantifiable  because  the 
recommendations  will  affect  an  undeterminable  number  of  future  programs 
transitioning  to  UUP  and  to  full-rate  production.  Appendix  I  notes  the  potential 
benefits  to  be  derived  from  implementing  the  recommendations. 

Summary  of  Recommendations.  We  recommend  that  the  Under  Secretary  of  Defense 
for  Acquisition  revise  acquisition  regulations  and  military  standards  to  provide 
additional  internal  controls  for  assessing  the  readiness  of  programs  to  enter  LRIP, 
including  a  required  milestone  review  before  entry  into  LRIP,  and  to  limit  the  number 
of  LRIP  units  produced  to  the  minimum  quantity  necessary  to  support  Initial 
Operational  Test  and  Evaluation  and  production  base  considerations. 

Management  Comments.  Management  comments  to  the  draft  report  were  provided 
for  the  Under  Secretary  of  Defense  for  Acquisition  by  the  Director,  Acquisition 
Program  Integration.  The  Director  partially  concurred  with  our  findings  and 
recommendations.  A  full  discussion  of  management  comments  and  audit  responses  is 
in  Part  II,  and  the  complete  texts  of  the  Director's  comments  as  well  as  additional 
comments  submitted  by  the  Office  of  the  Deputy  Assistant  Secretary  of  the  Air  Force 
(Acquisition)  and  the  Office  of  the  Army  Director  for  Combat  Service  Support  are  in 
Part  IV  of  this  report.  We  request  that  the  Director  reconsider  his  position  on 
noncurrences  and  provide  additional  comments  to  the  final  report  by  January  10,  1994. 
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Part  I  -  Introduction 


Background 


Low-Rate  Initial  Production  (LRIP),  as  defined  by  DoD  Instruction  5000.2, 
"Defense  Acquisition  Policies  and  Procedures,"  February  23,  1991,  is  the 
production  of  a  system  in  a  limited  quantity  to: 

o  Provide  articles  for  operational  test  and  evaluation, 

o  Establish  an  initial  production  base,  and 

o  Permit  an  orderly  increase  in  the  production  rate  so  that  full-rate 
production  will  start  after  successful  completion  of  operational  testing. 

The  purpose  of  UUP  is  to  verify  the  adequacy  of  the  manufacturing  or 
production  process,  confirm  the  stability  and  producibility  of  the  design, 
produce  units  for  operational  test  and  evaluation  of  system  capabilities,  and 
provide  information  to  support  Production  and  System  Configuration  Baselines. 

Planning  for  LRIP  should  begin  early  in  the  acquisition  process.  DoD 
Instruction  5000.2  requires  that  proposed  LRIP  quantities  be  determined  during 
Phase  I,  Demonstration  and  Validation,  and  that  milestone  decision  authorities 
approve  the  LRIP  quantity  at  the  Milestone  n,  Development  Approval,  decision 
point.  The  design,  testing,  and  production  preparation  necessary  to  support 
entry  into  LRIP  are  then  done  as  part  of  the  Phase  II,  Engineering  and 
Manufacturing  Development  (EMD),  effort.  Presently,  LRIP  is  part  of  the 
EMD  phase  of  the  acquisition  process  leading  to  Milestone  m,  Production 
Approval,  for  the  start  of  production  and  deployment.  Before  the  revised  5000 
series  acquisition  regulations  were  issued  in  February  1991,  LRIP  was 
frequently  a  separate  milestone  decision  point,  designated  Milestone  III  A.  No 
formal  DoD  requirement  existed  to  conduct  an  LRIP  milestone  review. 

DoD  guidance  for  transitioning  from  development  to  LRIP  to  full-rate 
production  is  provided  by  several  sources: 

o  Various  statutes  address  LRIP  and  operational  testing  requirements  as 
discussed  in  Appendix  A.  Of  particular  note,  United  States  Code,  title  10, 
section  2399  provides  that  a  major  Defense  acquisition  program  may  not 
proceed  beyond  LRIP  until  Initial  Operational  Test  and  Evaluation  (IOT&E)  is 
completed;  and  the  DoD  Director  of  Operational  Test  and  Evaluation  reports  to 
Congress  that  test  and  evaluation  were  adequate  and  that  the  results  of  test  and 
evaluation  confirm  that  the  items  or  components  tested  were  effective  and 
suitable  for  combat. 

o  DoD  Instruction  5000.2  requires  that  program  acquisition  strategies  be 
event-driven,  with  entry  into  LRIP  and  full-rate  production  based  on 
accomplishing  specific  program  results,  known  as  exit  criteria. 
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o  DoD  4245.7-M,  "Transition  from  Development  to  Production," 
issued  in  September  1985,  provides  guidance  on  minimizing  risks  associated 
with  transitioning  from  full-scale  development  to  production  through  timely 
accomplishment  of  prerequisites  in  design,  testing,  and  production  readiness. 

o  Military  Standard  1521-B,  "Technical  Reviews  and  Audits  for 
Systems,  Equipments,  and  Computer  Software,"  June  4,  1985,  identifies 
technical  reviews  and  audits  required  of  acquisition  programs  at  various  stages. 
As  systems  are  developed,  these  reviews  and  audits  provide  feedback 
concerning  the  suitability  of  system  hardware  and  software  design  and  the  risks 
associated  with  production  decisions.  Additional  DoD  guidance  concerning 
LRIP  is  in  Appendix  B. 

The  effective  planning  and  execution  of  the  LRIP  process  is  essential  for  a 
smooth  transition  to  economical  full-rate  production  of  systems  that  will  meet 
the  mission  requirements  of  planned  system  users. 

DoD  has  defined  "concurrency"  in  acquisition  strategies  as  the  degree  of 
overlap  between  the  development  and  production  processes  of  an  acquisition 
program.  Concurrency  is  most  prevalent  during  LRIP  where  engineering  and 
manufacturing  development  activities  continue  during  initial  production.  The 
DoD  rationale  for  high  concurrency  includes  providing  earlier  operational 
capability  when  the  need  is  time-urgent,  avoiding  technical  obsolescence,  and 
attaining  efficiencies  by  maintaining  the  production  process  and  work  force. 
DoD  has  also  recognized  that  low  concurrency  provides  for  greater  design 
maturity  that  increases  the  likelihood  of  meeting  system  requirements  and 
avoiding  retrofit  costs  to  make  production  articles  work  properly. 

The  1986  report  of  the  President's  Blue  Ribbon  Commission  on  Defense 
Management  (the  Packard  Commission  named  after  Chairman  David  Packard) 
and  the  subsequent  1989  Defense  Management  Report  by  the  Secretary  of 
Defense  emphasized  testing  prototype  hardware  to  reduce  concurrency  between 
development  and  production.  In  April  1990,  the  Under  Secretary  of  Defense 
for  Acquisition  (USD[A])  issued  a  report  to  Congress,  giving  a  detailed 
assessment  of  concurrency  and  associated  program  risk.  In  May  1992,  the 
USD(A)  stated  that,  as  a  result  of  the  break  up  of  the  Warsaw  Pact  and 
dissolution  of  the  Soviet  Union,  DoD  should  reduce  concurrency  in 
development  programs.  The  USD(A)  stated  specifically  that  less  risk  would  be 
accepted  in  acquisition  programs  than  had  been  accepted  in  the  past. 


Objectives 


Our  overall  objective  was  to  evaluate  the  effectiveness  of  DoD  use  of  LRIP  for 
major  Defense  acquisition  programs.  An  additional  objective  was  to  evaluate 
the  internal  management  controls  associated  with  the  LRIP  process.  The  audit 
is  one  of  a  series  of  reviews  to  assess  implementation  of  recommendations  in  the 
Secretary  of  Defense's  "Defense  Management  Report." 
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Scope 


To  satisfy  our  objectives,  we  examined  four  major  Defense  acquisition 
programs  that  either  were  in  LRIP  or  scheduled  to  enter  LRIP  before  January 
1993.  These  programs  were  judgmentally  selected  from  the  universe  of 
programs  meeting  the  criteria  for  major  Defense  acquisition  programs 
established  in  DoD  Instruction  5000.2,  "Defense  Acquisition  Management 
Policies  and  Procedures. "  Programs  included  one  each  from  the  Army  and  Air 
Force  and  two  from  die  Navy: 

o  Armv: 

Palletized  Load  System  (PLS) 
o  Navy: 

MK-50  Torpedo  System 

EA-6B  Aircraft  System 
o  Air  Force: 

Military  Strategic  and  Tactical  Relay  Satellite  Terminal 
(MILSTAR)  Program 

The  audit  was  performed  between  April  1992  and  January  1993  and  included  a 
review  of  LRIP  data  and  information  from  1983  to  January  1993.  Our 
examination  resulted  in  an  assessment  of  criteria  established  for  initial  and 
follow-on  LRIP  decisions  in  program  acquisition  strategies  and  compliance  with 
these  criteria.  We  also  assessed  support  for  identified  LRIP  quantities  and 
decision  schedules,  the  approval  process  for  LRIP  decisions  and  contract 
awards,  proper  utilization  of  development  and  production  funding,  support  for 
proceeding  to  full-rate  production,  and  corrective  actions  on  related  LRIP 
findings  in  prior  audits. 

We  examined  information  and  data  in  acquisition  plans,  operational 
requirements  documents,  contract  files,  test  and  evaluation  master  plans, 
developmental  and  operational  test  reports,  and  technical  reviews  and  audits 
performed  to  support  progress  toward  LRIP  or  full-rate  production.  We 
interviewed  personnel  responsible  for  program  management,  procurement, 
testing,  and  contract  administration,  as  well  as  Defense  plant  representatives  and 
contractor  personnel  to  determine  and  evaluate  the  policies  and  procedures 
followed  in  the  LRIP  process.  We  also  examined  the  LRIP  information  given 
to  senior  acquisition  management  in  the  "Defense  Acquisition  Executive 
Summary"  and  other  reports. 

To  supplement  our  audit  coverage,  we  also  identified  LRIP-related  issues 
included  in  Inspector  General,  DoD,  and  General  Accounting  Office  (GAO) 
reports  on  four  additional  systems.  These  systems  included  the  Navy's 
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Airborne  Self-Protection  Jammer  (ASPJ)  and  the  Air  Force's  C-17  Aircraft, 
Sensor  Fused  Weapon  (SFW)  System,  and  Joint  Tactical  Information 
Distribution  System  (JTIDS).  Appendix  C  provides  a  synopsis  of  these  reports. 

This  program  results  audit  was  made  in  accordance  with  auditing  standards 
issued  by  the  Comptroller  General  of  the  United  States,  as  implemented  by  the 
Inspector  General,  DoD,  and  included  necessary  tests  of  internal  controls. 
Appendix  J  lists  activities  visited  or  contacted. 


Internal  Controls 

We  evaluated  the  adequacy  of  internal  controls  over  the  LRIP  process.  As  part 
of  our  evaluation,  we  assessed: 

o  Statutory  and  DoD  regulatory  guidance  on  the  LRIP  process; 

o  Military  Department  implementing  procedures  and  compliance  with 
regulations;  and 

o  Oversight  of  the  LRIP  process  by  the  Office  of  the  Secretary  of 
Defense  (OSD)  and  the  DoD  Components. 

The  audit  identified  material  internal  control  weaknesses,  as  defined  by  Public 
Law  97-255,  Office  of  Management  and  Budget  Circular  A-123,  and  DoD 
Directive  5010.38.  OSD  and  Military  Department  guidance  and  oversight  for 
the  LRIP  process  were  not  adequate  to  ensure  that  major  Defense  acquisition 
programs  completed  the  prerequisites  in  design,  testing,  and  preparation  for 
production  necessary  to  reduce  risk  when  transitioning  into  LRIP.  Additionally, 
increased  OSD  and  Military  Department  oversight  was  needed  to  ensure  that 
LRIP  units  were  limited  to  the  minimum  quantities  required  for  testing  and 
production  purposes  and  to  ensure  that  all  prerequisites  were  met  before  making 
full-rate  production  and  deployment  decisions. 

Implementation  of  the  recommendations  for  Findings  A  and  B  will  correct  these 
weaknesses.  The  monetary  benefits  of  implementing  these  recommendations 
were  not  readily  quantifiable  because  the  recommendations  will  affect  an 
undeterminable  number  of  future  programs  transitioning  to  LRIP  and  to  foil-rate 
production.  A  copy  of  the  report  is  provided  to  senior  officials  responsible  for 
internal  controls  in  the  Office  of  the  Secretary  of  Defense  and  the  Military 
Departments. 
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Prior  Audits  and  Other  Reviews 


Since  1989,  the  Inspector  General,  DoD,  had  issued  eight  reports  and  the  GAO 
had  issued  six  reports  addressing  issues  related  to  low-rate  initial  production. 
These  reports  are  synopsized  in  Appendix  C.  During  the  course  of  this  audit, 
we  issued  reports  on  the  Navy  EA-6B  Aircraft  Remanufacture  and  Air  Force 
MIT  .STAR  Terminal  programs,  as  well  as  a  memorandum  to  the  USD(A)  on  the 
Army  PLS.  Our  reports  and  memorandum  addressed  time-sensitive  concerns 
relating  to  program  readiness  for  upcoming  production  decisions.  Appendix  D 
synopsizes  these  reports. 
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Part  II  -  Findings  and  Recommendations 
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Finding  A.  Readiness  for  Low-Rate 
Initial  Production 

All  seven  of  the  major  Defense  acquisition  programs  reviewed  entered 
LRIP  without  completing  at  least  some  prerequisites  in  design,  testing, 
and  preparation  for  production.  Premature  entry  into  LRIP  was  caused 
by  inadequacies  in  the  milestone  review  process,  regulations,  and  policy 
guidance  for  LRIP.  Also,  program  planning  was  done  when  urgency  to 
meet  threats  justified  highly  concurrent  development  and  production 
efforts.  As  a  result,  the  Government  incurred  significant  program  risk 
from  systems  entering  LRIP  when  their  designs  were  not  stable  and  the 
readiness  of  production  processes  was  not  verified. 


Background 


Three  critical  decision  points  preceed  entry  into  LRIP: 

o  the  Milestone  II,  Development  Approval,  which  approves  the  program 
acquisition  strategy  of  LRIP  and  LRIP  quantities; 

o  LRIP  long-lead  funding  approval;  and 

o  LRIP  approval. 

At  the  Milestone  II  decision  point,  DoD  Instruction  5000.2  requires  that  the 
Director,  Operational  Test  and  Evaluation,  determines  the  quantities  of  LRIP 
articles  required  for  operational  testing  of  a  Major  Defense  Acquisition 
Program.  Additionally,  Change  1  to  DoD  Instruction  5000.2,  dated 
February  26,  1993,  states  that  authority  to  proceed  with  LRIP  may  require  a 
separate  program  review  and  milestone  decision  authority  approval  at  a  point 
specified  in  the  Milestone  II  decision.  However,  DoD  Instruction  5000.2  does 
not  contain  direction  on  determining  the  LRIP  quantities  to  be  produced  and  exit 
criteria  for  entry  into  LRIP  and  subsequent  LRIP  production  lots. 
Considerations  in  determining  LRIP  quantities  are  only  provided  for  Naval 
vessels  and  satellites,  which  can  have  unique  total  quantities  and  production 
periods.  These  considerations  include  complexity  of  the  system,  total  number  to 
be  procured,  length  of  the  production  period,  industrial-base  implications,  and 
acquisition  strategy  most  advantageous  to  the  Government. 

Obligation  of  long-lead  funding  to  support  entry  into  LRIP  is  the  second  critical 
decision  point  associated  with  LRIP.  The  long-lead  funding  decision  point 
represents  the  commitment  of  funds  to  initiate  production-related  activities.  The 
April  1990  USD(A)  report  to  Congress  on  concurrency  guidelines  proposed  that 
clear  exit  criteria  be  established  for  initiation  of  long-lead  funding  for  LRIP  and 
that  the  decision  to  commit  funds  be  supported  by  operational  test  assessments. 
However,  DoD  Instruction  5000.2  and  other  acquisition  guidance  do  not 
establish  policy  for  the  commitment  of  long-lead  funding  for  LRIP. 
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The  third  critical  decision  point  associated  with  LRIP  is  the  approval  of  entry 
into  LRIP.  As  stated,  the  1993  change  to  DoD  Instruction  5000.2  suggests,  but 
does  not  require,  a  program  review  and  milestone  decision-authority  approval  of 
proceeding  into  LRIP.  The  new  guidance  in  DoD  Instruction  5000.2  also 
suggests  that  exit  criteria  be  established  that,  when  successfully  passed,  allow 
the  program  office  to  expand  activities  or  commitments  during  an  acquisition 
phase.  Long-lead  procurement  funding  and  LRIP  are  examples  of  such 
commitments.  Finally,  the  new  guidance  states  that  additional  activities  or 
program  reviews  are  triggered  by  failure  to  meet  exit  criteria  established  for 
proceeding  into  LRIP. 

In  summary,  LRIP  is  a  critical  element  in  the  acquisition  strategy  for  a  weapon 
system.  LRIP  allows  the  contractor  to  start  the  system  production  line 
concurrently  with  ongoing  engineering  development.  Therefore,  the  acquisition 
strategy,  implementing  acquisition  plan,  and  acquisition  management  process 
should  ensure  that  the  decision  to  begin  LRIP  is  based  upon  demonstrated 
technical  and  performance  accomplishments,  not  schedule  or  fiscal 
considerations.  DoD  has  emphasized  the  use  of  event-driven  acquisition 
strategies  so  program  prerequisites  are  accomplished  timely  and  in  the 
appropriate  sequence. 

Initiative  to  Reduce  Program  Risk.  On  April  17,  1990,  the  USD(A)  issued 
the  "Report  on  Guidelines  for  Determining  the  Degree  of  Risk  Appropriate  for 
the  Development  of  Major  Defense  Acquisition  Systems,  and  Assessing  the 
Degree  of  Risk  Associated  with  Various  Degrees  of  Concurrency;  and 
Concurrency  in  Major  Acquisition  Programs."  This  report  responded  to 
Section  801  of  the  National  Defense  Authorization  Act  for  Fiscal  Years  1990 
and  1991,  which  required  establishing  guidelines  for: 

o  Determining  the  degree  of  concurrency  that  is  appropriate  for  the 
development  of  major  Defense  acquisition  systems;  and 

o  Assessing  the  degree  of  risk  associated  with  degrees  of  concurrency. 

The  1990  report  provided  guidelines  for  assessing  concurrency  and  risk  and 
proposed  that  these  guidelines  be  in  the  revision  to  the  DoD  Directive  5000.1, 
"Major  and  Non-Major  Acquisition  Programs,"  later  issued  in  February  1991. 
The  proposed  guidelines  were  to  help  establish  acquisition  strategies  for 
acquisition  programs  that  explicitly  link  milestone  decisions  to  demonstrated 
accomplishments.  The  guidelines  specific  to  the  LRIP  decision  included: 

o  Ensuring  that  the  acquisition  strategy  will  provide  confidence  that  a 
stable  design  exists  before  the  program  moves  into  LRIP  (LRIP  validates  the 
production  process  and  the  design  must  be  stable  at  this  point); 

o  Establishing  clear  exit  criteria  for  initiation  of  long-lead  funding  for 
LRIP  and  for  entry  into  LRIP; 

o  Ensuring  that  all  development  testing  is  properly  time-phased  so 
technical  problems  are  highlighted  before  they  become  critical;  and 
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o  Ensuring  that  engineering  development  articles,  which  usually  will  be 
used  to  perform  the  testing  upon  which  initial  production  decisions  will  be 
made,  are  representative  of  the  production  configuration. 

In  addition  to  providing  specific  guidelines  for  the  LRIP  decision,  the  report 
proposed  that  program  concurrency  risk  be  reduced  through  "aggressive"  use  of 
.  prototyping  and  testing  to  identify  and  remedy  problems  well  before  production 
starts.  The  Packard  Commission  was  cited  as  the  source  for  this  proposal.  We 
are  separately  evaluating  the  use  of  prototyping  in  our  "Audit  of  DoD  Use  of 
Prototyping  Acquisition  Strategies  for  Major  Defense  Acquisition  Programs" 
(Project  No.  2AE-0051). 

Revision  of  DoD  Guidance.  When  the  revised  DoD  Directive  5000.1  and  the 
implementing  DoD  Instruction  5000.2  were  issued  in  February  1991,  the 
guidance  for  LRIP  was  very  general.  The  revisions  did  not  include  the  specific 
LRIP-related  guidance  as  proposed  in  the  April  1990  report. 

In  addition  to  proposing  concurrency  guidelines,  the  USD(A)  report  defined 
three  levels  of  program  concurrency: 

o  Low  -  Program  will  have  completed  essentially  all  IOT&E  before 
entering  production  (either  LRIP  or  beyond  LRIP); 

o  Moderate  -  Program  proceeds  into  LRIP  with  only  part  of  the  early 
IOT&E  completed;  and 

o  High  -  Programs  proceeds  into  LRIP  before  significant  IOT&E  is 
completed. 

Based  on  the  results  of  our  review  of  program  test  accomplishment  before 
LRIP,  we  believe  all  eight  programs  we  reviewed  involved  moderate  or  high 
levels  of  concurrency. 


Planning  and  Preparation  for  Low-Rate  Initial  Production 


The  planning  and  preparation  for  LRIP  decisions  did  not  provide  for  fulfilling 
the  design,  testing,  and  production  preparation  prerequisites  established  under 
DoD  Instruction  5000.2,  "Defense  Acquisition  Management  Policies  and 
Procedures";  Military  Standard  1521-B,  "Technical  Reviews  and  Audits  for 
Systems,  Equipments,  and  Computer  Software";  and  DoD  4245. 7-M, 
"Transition  from  Development  to  Production,"  for  transition  into  LRIP. 
Shortfalls  in  program  accomplishment  and  documentation  in  design,  testing,  and 
manufacturing  areas  critical  to  support  entry  into  LRIP  included: 

o  Essential  testing,  assessments,  and  technical  reviews  not  performed; 

o  Significant  problems,  identified  in  testing  or  technical  reviews, 
scheduled  for  resolution  after  entry  into  LRIP  decision; 
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o  Documentation  from  technical  reviews  and  from  developmental, 
reliability,  and  operational  testing,  not  planned  to  be  available  to  support 
scheduled  LRIP  decisions. 

Appendix  E  summarizes  the  LRIP  deficiencies  for  seven  of  the  eight  major 
Defense  acquisition  programs  included  in  this  audit.  Discussion  concerning  the 
eighth  system,  the  Air  Force  SFW,  is  limited  to  Finding  B,  Low-Rate  Initial 
Production  Quantities  and  Commitments. 

Design.  Our  review  of  contract  specifications  and  program  plans  and  progress 
in  the  technical  reviews  and  audits  required  by  Military  Standard  1521-B 
showed  that  programs  entered  LRIP  with  unresolved  design  problems  or  before 
completion  of  technical  reviews  to  support  the  LRIP  decision. 

Design  Problems.  We  found  that  six  of  the  seven  programs  we 
reviewed  for  LREP  readiness  entered  or  planned  to  enter  LRIP  with  unresolved 
design  problems.  The  Navy  MK-50  Torpedo  and  the  Air  Force  MILSTAR 
Terminal  programs  are  examples  of  this  problem. 

The  MK-50  Torpedo  Program  issued  its  first  LRIP  contract  with  48  known, 
unresolved  design  problems,  which  were  noted  by  the  critical  design  review. 
Additionally,  16  percent  of  the  hardware  documentation  in  the  Technical  Data 
Package  was  incomplete  at  the  time  of  the  LRIP  decision  for  MK-50  in  May 
1989.  To  complete  the  technical  data  package,  work  on  the  design  problems 
was  necessary  during  LRIP.  MK-50  subsequently  experienced  program  delays; 
an  additional  LRIP  buy  was  required  due  to  the  correction  of  technical  problems 
as  discussed  in  Finding  B. 

The  Air  Force  MILSTAR  Terminal  Program,  as  noted  in  the  Inspector  General, 
DoD,  Report  No.  93-084,  "Air  Force  Strategic  and  Tactical  Relay  Satellite 
Terminal  Program,"  April  13,  1993,  was  allowed  to  begin  LRIP  of  MILSTAR 
command  post  terminals  in  December  1989  even  though  the  development 
contractor  had  not  completed  terminal  design.  Because  of  the  amount  of  testing 
and  development  yet  to  be  done,  LRIP  contracts  were  awarded  with  "B"  level 
specifications  (required  terminal  performance  capabilities)  rather  than  "C"  level 
specifications  (design  drawings).  The  contractor  was  not  scheduled  to  deliver 
some  production  drawings  until  January  15,  1993,  which  was  less  than  2  months 
before  the  scheduled  production  and  deployment  decision  planned  by  the 
Program  Executive  Officer. 

As  of  November  18,  1992,  the  contractor  was  still  completing  terminal  design. 
This  ongoing  work  included  correction  of  52  open  Category  II  Product  Quality 
Deficiency  Reports.  Category  II  deficiencies  are  defined  by  Air  Force 
Technical  Order  00-35D-54  as  deficiencies  attributable  to  errors  in 
workmanship  or  nonconformance  to  specifications,  drawing  standards,  or  their 
technical  requirements  that  can  result  in  hazardous  or  unsafe  conditions  for 
individuals  using,  maintaining  or  depending  on  the  product  or  prevent 
performance  of  the  product's  tactical  or  strategic  function.  Additionally, 


11 


Finding  A.  Readiness  for  Low-Rate  Initial  Production 


development  work  was  planned  until  September  1994.  The  additional 
development  work  was  necessary  to  bring  the  capacity  of  the  terminal's  central 
processing  unit  to  within  5  percent  of  the  capacity  required  by  the  Operational 
Requirements  Document,  May  28,  1992. 

Programs  entering  LRIP  without  a  mature,  stable  design,  frequently 
experienced  production-related  problems  and  delays  that  introduced  the  need  to 
make  additional  LRIP  awards  to  preclude  the  costs  associated  with  a  break  in 
production. 

Required  Reviews.  We  determined  that  the  Navy  EA-6B 
Remanufacture  program  and  the  Air  Force  C-17  program  had  not  completed 
required  critical  design  reviews  (CDRs)  as  a  prerequisite  to  LRIP  decisions.  In 
addition,  the  Navy  EA-6B  had  not  completed  the  software  test  readiness  review 
(TRR).  Both  Military  Standard  1521-B  and  DoD  4245.7-M  state  that  CDRs 
and  software  TRRs  are  normally  performed  during  Phase  II,  Engineering  and 
Manufacturing  Development,  of  the  acquisition  process.  Each  review  provides 
critical  information  as  to  die  suitability  of  planned  hardware  and  software 
system  configurations  for  production  and  the  completeness  of  system  design. 
The  CDR  is  defined  in  DoD  Instruction  5000.2  as  a  review  to: 

o  Determine  whether  the  detailed  design  of  a  system  meets  the 
performance  and  engineering  requirements  of  the  development  specification; 

o  Establish  the  detailed  design  compatibility  among  the  end  item 
and  other  items  of  equipment,  facilities,  computer  programs,  and  personnel; 

o  Assess  producibility  and  risk  areas;  and 

o  Review  the  preliminary  product  specifications.  CDRs  are  to 
be  conducted  on  both  hardware  and  software  portions  of  a  system. 

The  TRR  is  defined  by  Military  Standard  1521-B  as  a  review  of  computer 
software  configuration  items  to  determine  contractor  readiness  to  begin  formal 
software  testing.  The  impact  of  software  design  changes,  software  test 
resources,  known  software  problems,  and  limitations  to  test  software  are  among 
the  factors  assessed. 

In  Inspector  General,  DoD,  Report  No.  93-039,  "Audit  Report  on  the  Low-Rate 
Initial  Production  of  the  EA-6B  Program,"  December  18,  1992,  we  reported 
that  the  Government  would  be  unable  to  ensure,  before  the  LRIP  decision,  that 
all  design  areas  were  adequately  examined,  that  design  weaknesses  were 
identified,  and  that  solutions  for  design-related  issues  were  available.  This 
problem  occurred  because  hardware  and  software  CDRs  for  the  remanufactured 
EA-6B  aircraft  (the  advanced  capability  onboard  system  fully  integrated  with  the 
vehicle  enhancement  program  and  the  avionics  improvement  program)  were 
scheduled  to  start  after  the  scheduled  LRIP  decision  in  September  1992.  In 
addition,  we  also  reported  that  the  Government  could  not  fully  assess  the 
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contractor's  ability  to  test  software  adequately  because  the  TRRs  for  the  receiver 
processor  group  (a  portion  of  the  advanced  capability  onboard  system)  and  the 
overall  remanufactured  aircraft  would  not  be  done  until  after  the  scheduled 
September  1992  LRIP  decision. 

In  Inspector  General,  DoD,  Report  No.  91-007,  "Audit  Report  on  Selected 
Acquisition  Actions  on  the  C-17  Aircraft,"  November  2,  1992,  we  stated  that 
the  Air  Force  exercised  an  FY  1989  contract  option  to  buy  four  LRIP  aircraft 
(in  addition  to  two  other  LRIP  aircraft  that  were  already  on  contract)  before 
completing  the  CDR  of  all  mission  computer  software.  The  C-17  program 
office  considered  the  CDR  to  be  complete  because  the  top-level  software  design 
of  the  mission  computer  was  reviewed  during  the  April  1989  CDR  and  the 
contractor  had  taken  management  actions  to  ensure  completion  of  the  detail 
design  of  the  mission  computer  software.  Military  Standard  1521-B  states, 
however,  that  the  CDR  should  be  a  formal  review  of  the  detail  software  design. 
The  audit  report  stated  that  when  the  CDR  was  conducted,  detail  designs  were 
available  for  only  about  60  percent  of  all  software  in  the  mission  computer. 

Testing.  Our  review  of  testing  documentation  showed  that  she  of  the 
seven  programs  we  reviewed  for  LRIP  readiness  experienced  shortfalls  in 
accomplishment  and  documentation  of  developmental  or  operational  tests  before 
entering  LRIP.  These  test-related  shortfalls  included: 

o  Unresolved  deficiencies  in  the  demonstrated  ability  of  systems  to  meet 
technical  and  operational  mission  requirements;  and 

o  Developmental  and  operational  testing  occurring  before  LRIP  that  was 
reduced  or  limited  and  as  a  result  did  not  provide  the  documentation  needed  to 
make  an  informed  LRIP  decision. 

The  April  1990  USD(A)  report  to  Congress  concluded  that  the  determination  of 
whether  a  program  is  ready  to  enter  LRIP  must  "be  based  upon  the  totality  of 
component,  subsystem  and  system  testing  that  is  done  (or  not  done),  and  the 
results  of  this  testing"  (underscoring  added  for  emphasis).  In  fact,  the  report 
bases  determinations  concerning  the  degree  of  concurrency  in  a  program  on  the 
amount  of  IOT&E  completed  that  supports  the  entry  into  LRIP.  We  consider 
essential  the  internal  control  recommended  in  the  USD(A)  report  concerning 
performance  of  independent  operational  assessments  before  committing  funds  to 
enter  LRIP,  as  discussed  below. 

Developmental  Testing.  Four  of  the  seven  programs  reviewed  for 
LRIP  readiness  had  significant  deficiencies  identified  through  developmental 
testing  that  were  not  resolved  (specific  fixes  identified  by  contractor)  before 
entering  LRIP.  Additionally,  four  of  seven  programs  had  reduced  or  limited 
developmental  testing  before  die  LRIP  decision. 

Unresolved  Deficiencies.  The  Army's  PLS  is  one  example  of  a 
system  having  unresolved  deficiencies  from  developmental  testing  when  going 
into  LRIP.  Before  its  LRIP  decision  in  September  1990,  the  PLS  Program 
conducted  Preproduction  Qualification  Testing  with  three  competitors  using  PLS 
prototypes.  No  competitor  achieved  the  truck  reliability  requirement  of 
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1,600  mean  miles  between  hardware  mission  failures  during  this  technical  test. 
In  fact,  the  contractor  selected  to  produce  the  PLS  only  achieved  a  raw  score  of 
550  mean  miles.  After  an  assessment  by  the  Army  Materiel  Systems  Analysis 
Activity,  the  score  was  raised  to  approximately  900  mean  miles,  still  well  below 
the  requirement.  This  technical  problem  should  have  been  addressed  in 
additional  testing  before  entering  LRIP.  However,  no  additional  technical 
.  testing  was  performed  before  the  LRIP  decision  and  the  production  contract 
award  in  September  1990.  The  next  scored  technical  test  events,  the 
Shakedown  Test  and  the  Production  Qualification  Test,  occurred  8  and 
13  months,  respectively,  after  the  LRIP  decision  and  production  award. 

The  Navy  EA-6B  Remanufacture  program  also  was  scheduled  to  begin  LRIP 
with  unresolved  deficiencies  from  developmental  testing.  In  Inspector  General, 
DoD,  Report  No.  93-039,  we  stated  that  developmental  testing  on  the  advanced 
capability  onboard  system  by  the  Naval  Air  Warfare  Center  between  January 
and  May  1992  found  46  Part  I  (safety  of  flight  or  inability  to  complete  primary 
mission)  deficiencies.  While  the  developmental  test  report  issued  by  the  Naval 
Air  Warfare  Center  on  May  14,  1992,  states  that  both  the  Receiver  Processor 
Group  and  the  ALQ-149  Communications  Jammer  will  be  satisfactory  after 
correction  of  the  Parti  deficiencies,  identification  of  fixes  for  32  of  the 
46  deficiencies  was  not  scheduled  until  after  the  LRIP  decision. 

Reduced  or  Limited  Testing.  Inspector  General  Report 
No.  93-039  stated  that  the  Navy  had  allowed  a  significant  reduction  in  the 
planned  reliability  development  testing  to  be  completed  by  the  contractor  to 
support  the  EA-6B  LRIP  decision.  Originally,  the  engineering  and 
manufacturing  development  contract  required  between  1,500  hours  and 
2,000  hours  of  testing  on  each  of  13  unique  weapon  replaceable  assemblies 
making  up  the  receiver  processor  group.  Contract  Specification  Change 
Notice  3,  submitted  by  Grumman  Corporation  in  November  1989,  reduced  the 
contract  requirement  to  400  hours  on  4  weapon  replaceable  assemblies  deemed 
to  be  system  representative.  This  reduction  in  testing  to  support  the  LRIP 
decision  resulted  from  delays  related  to  design  problems  experienced  by  Litton 
Corporation,  die  subcontractor  responsible  for  providing  the  receiver  processor 
group.  As  of  July  1992,  the  Navy  had  not  formally  accepted  the  change  but  had 
operated  as  if  there  was  acceptance. 

DoD  4245 .7-M  states  that  reliability  development  testing  reduces  the  risk  of 
allowing  systems  with  poor  reliability  to  transition  from  development  to 
production  and  that  this  testing  should  be  completed  before  the  initial  production 
decision.  This  manual  also  states  that  test  stability,  defined  as  the  absence  of 
failures  in  development  testing  of  a  stable  design,  is  essential  to  reduce  risk  in 
the  transition  from  development  to  production. 

Inspector  General  Report  No.  93-039  also  stated  that  developmental  testing  of 
the  overall  EA-6B  remanufacture  configuration  was  not  planned  to  support  the 
scheduled  September  1992  LRIP  decision.  Testing  of  the  overall  configuration 
could  not  occur  until  the  three  separate  program  segments  (advanced  capability 
onboard  system,  vehicle  enhancement  program,  and  avionics  improvement 
program)  are  merged  into  one  aircraft  in  January  1994. 
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The  MILSTAR  terminal  program,  as  noted  in  GAO  Report  No.  GAO/C- 
NSIAD-90-28  (OSD  Case  No.  8177),  "Military  Satellite  Communications: 
Issues  Associated  With  DoD’s  MILSTAR  Terminal  Program,"  May  23,  1990, 
provides  another  example  of  a  system  proceeding  to  LRIP  with  less  testing  than 
originally  planned.  The  GAO  concluded  that  the  June  1989  decision  authorizing 
the  Air  Force  to  proceed  into  low-rate  terminal  production  of  command  post 
terminals  was  based  on  less  test  data  than  originally  planned,  thus  increasing  the 
risk  that  design  changes  could  occur. 

According  to  the  GAO  report,  data  from  field  development  tests  of  EMD 
terminals  were  limited  due  to  defects  in  the  terminal  radome  (relates  to  the 
terminal  antenna  function)  and  nonavailability  of  test  aircraft. 

Operational  Assessments.  Operational  assessments  are  evaluations  of 
operational  effectiveness  and  suitability  made  by  an  independent  operational  test 
activity,  with  user  support  as  required,  on  other  than  production  systems. 
Operational  assessments  differ  from  operational  test  and  evaluation  because 
production  systems  are  not  required;  rather,  assessments  use  technology 
demonstrators,  prototypes,  or  engineering  development  models  that  should  be 
production-representative.  Aso,  operational  assessments  can  rely  more 
extensively  on  computer  modeling,  simulation,  and  analysis  of  program 
information.  The  April  1990  USD(A)  report  to  Congress  stated  that  decisions 
to  commit  funds  for  LRIP  are  supported  by  operational  assessments;  however, 
we  found  no  requirement  to  perform  the  assessment  before  committing  funds  is 
presently  in  DoD  Instruction  5000.2.  Our  review  of  operational  test 
documentation  showed  that  four  of  the  seven  programs  examined  had  only 
limited  operational  assessments  to  support  entering  LRIP. 

In  Inspector  General  Report  No.  93-039,  we  disclosed  that  an  overall 
operational  assessment  of  the  EA-6B  Remanufactured  aircraft  had  not  been 
performed  in  support  of  the  LRIP  decision.  Operational  testing  to  support  the 
LRIP  decision,  as  the  developmental  testing  discussed  above,  was  limited  to  the 
advanced  capability  onboard  system.  In  addition,  operational  testing  used 
engineering  development  models  and  not  production-representative  units. 
Operational  testing  was  further  limited  by  the  results  of  developmental  testing. 
Personnel  at  the  Naval  Air  Warfare  Center,  Patuxent  River,  Maryland,  a  Navy 
test  activity,  stated  that,  based  on  developmental  test  results,  the  following 
processing  categories  were  not  considered  operationally  mission-suitable  and 
should  not  be  expected  to  perform  at  that  level  during  operational  tests: 

o  radar  warning  signal  processing  and  display, 

o  identification  functions, 

o  reactive  assignments  to  complex  emitters,  and 

o  ALQ-149  radar  and  communication  functions. 

In  summary,  the  operational  testing  on  the  advanced  capability  onboard  system 
did  not  provide  an  adequate  basis  for  proceeding  with  the  planned  September 
1992  LRIP. 
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The  MK-50  torpedo  is  another  system  that  provides  an  example  of  limited 
operational  assessment  before  the  LRIP  decision.  The  MK-50  was  not  tested 
against  a  fast-deep  target  before  entry  into  LRIP  in  March  1989.  A  program 
was  established  in  February  1986  to  develop  a  fast-deep  target  for  testing  and 
was  to  be  completed  in  1987  at  a  cost  of  $4.5  million.  However,  the  program 
had  many  technical  difficulties,  and  the  program  costs  increased  to 
.  $34.5  million.  In  February  1992,  the  target  program  was  terminated.  The 
Navy  determined  that  the  MK-50  would  be  tested  using  simulations  rather  than 
a  fast-deep  target. 

JTIDS  is  another  example  of  limited  testing  before  the  LRIP  decision. 
According  to  GAO  Report  No.  GAO/NSIAD-93-16  (OSD  Case  No.  8996), 
"Military  Communications:  Joint  Tactical  Information  Distribution  System 
Issues,"  November  12,  1992,  the  Air  Force  Operational  Test  and  Evaluation 
Center  (AFOTEC)  performed  an  operational  assessment  of  JTIDS  Class  2 
terminals  from  April  to  May  1989  to  support  an  LRIP  decision  for  the 
terminals.  AFOTEC  concluded  that  some  terminals'  operations  had  improved 
since  the  first  multi-Service  operational  test  (conducted  by  the  Air  Force)  in 

1987.  However,  the  number  of  operating  hours  were  insufficient  to  establish 
any  confidence  in  the  test  results.  Therefore,  AFOTEC  could  not  make  an 
adequate  assessment  of  the  terminals. 

ASPJ  provides  a  final  example  of  excessive  limitations  to  operational  testing 
before  entering  LRIP.  IG,  DoD,  Audit  Report  No.  90-066,  "Hotline 
Allegations  Regarding  the  Milestone  HIA  Production  Decision  for  the  Airborne 
Self-Protection  Jammer  Program,"  May  10,  1990,  reported  on  the  extent  of 
operational  test  and  evaluation  on  the  ASPJ.  Three  IOT&E  efforts  (Phases  HA, 
HB,  and  IIC)  were  conducted  between  June  and  November  1988,  before  the 
ASPJ  LRIP  decision  in  August  1989.  The  purpose  of  the  IOT&E  efforts  was  to 
resolve  or  partially  resolve  critical  operational  test  issues  before  making  ASPJ 
production  decisions.  The  ASPJ  Test  Plan  listed  20  critical  operational  test 
issues  that  required  resolution. 

None  of  the  three  test  efforts  was  conducted  with  a  production-representative 
ASPJ  in  an  operationally  realistic  threat  environment.  The  first  of  these  test 
efforts,  Phase  HA,  was  conducted  in  June  1988  by  the  Commander,  Naval 
Operational  Test  and  Evaluation  Force  (OPTEVFOR),  and  the  Commander, 
AFOTEC.  A  purpose  of  the  test,  done  when  the  engineering  and  development 
model  of  ASPJ  was  at  an  early  stage  of  development,  was  to  assess  the  potential 
effectiveness  and  suitability  of  the  ASPJ  and  support  Navy  and  Air  Force 
decisions  to  buy  14  ASPJ  production  verification  units.  Neither  test 
organization  was  able  to  fully  assess  ASPJ's  suitability  because  of  limitations  to 
the  scope  of  testing.  The  test  organizations  concluded  that  the  ASPJ  was 
potentially  operationally  effective  and,  therefore,  supported  a  recommendation 
to  buy  the  14  production  verification  units. 

The  Commander,  OPTEVFOR,  conducted  a  portion  of  Test  Phase  IIC  in  July 

1988,  and  the  Commander,  AFOTEC,  conducted  the  remainder  of  the  testing  in 
November  1988.  Test  phase  IIB  was  conducted  in  September  1988  by  the 
Commander,  AFOTEC.  Test  limitations  prevented  Navy  and  Air  Force  test 
personnel  from  assessing  9  of  the  20  critical  operational  issues  and  from  fully 
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resolving  the  remaining  11  critical  operational  issues  during  operational  tests. 
As  a  result,  operational  testers  were  prevented  from  fully  assessing  whether 
ASPJ  would  sufficiently  and  reliably  increase  aircraft  survivability  against 
enemy  defense  systems,  thereby  providing  an  operational  assessment  of 
operational  effectiveness  and  suitability  before  the  August  1989  LRIP  decision. 

Three  of  the  four  programs  we  cited  as  having  had  limited  operational 
assessments  before  LRIP  had  already  entered  LRIP  at  the  time  of  our  review. 
These  programs  were  the  MK-50,  ASPJ,  and  the  JTIDS  Class  2  and 
2H  terminals.  The  difficulties  these  programs  experienced  in  LRIP  resulting 
from  testing  shortfalls  and  unresolved  operational  issues  are  discussed  in 
Finding  B.  For  the  fourth  system,  EA-6B,  entry  into  LRIP  was  delayed  from 
September  1992  to  allow  for  additional  operational  testing  and  design  work. 
This  delay  was  responsive  to  the  intent  of  our  recommendations  in  our  draft  of 
Inspector  General,  DoD,  Report  No.  93-039;  this  draft  report  was  issued 
August  14,  1992. 

Manufacturing.  Our  review  of  program  office  documentation  showed 
shortfalls  in  meeting  requirements  provided  by  Military  Standard  152 1-B, 
"Technical  Review  and  Audits  for  Systems,  Equipments,  and  Computer 
Software,"  for  performance  of  production  readiness  reviews  before  entering 
LRIP.  In  addition,  EMD  models  of  systems  were  incomplete  and  not 
reasonably  representative  of  production  articles  for  purposes  of  operational 
testing  and  assessment. 

Production  Readiness  Reviews.  DoD  Instruction  5000.2  defines  a 
system  as  ready  for  production  when  the  producibility  of  the  production  design 
and  the  managerial  and  physical  preparations  necessary  for  initiating  and 
sustaining  a  viable  production  effort  have  progressed  so  that  a  production 
commitment  can  be  made  without  incurring  unacceptable  risk.  A  Production 
Readiness  Review  (PRR)  is  a  technical  review  of  the  completeness  and 
producibility  of  the  product  design  and  the  planning  and  preparation  for 
production.  A  PRR  typically  addresses  product  design,  industrial  resources, 
production  engineering  and  planning,  materials  and  purchased  parts,  and  quality 
assurance.  A  PRR  must  be  satisfactorily  accomplished  before  a  production 
decision.  PRRs  can  identify  potential  deficiencies  in  a  contractor's  program 
management,  logistics  support,  funding,  and  manufacturing  programs  that  could 
adversely  affect  production  schedules,  costs,  and  technical  performance. 

Military  Standard  152 1-B  requires  these  reviews  to  be  done  incrementally  with 
the  first  within  90  days  of  the  critical  design  review.  Subsequent  reviews  are  to 
be  at  least  yearly  thereafter  until  the  program  passes  Milestone  III,  Production 
Approval.  We  found  that  four  of  seven  programs  in  our  review  did  not  meet 
the  Military  Standard  requirements  for  PRRs  because  the  reviews  were  either 
not  performed  or  unduly  limited  in  coverage. 

Review  Performance.  No  PRRs  were  performed  for  PLS 
program  before  entry  into  LRIP.  Although  PLS  is  based  on  the  integration  of 
present  technology,  it  is  a  unique  system.  Major  modifications  to  existing 
technology  were  required  for  PLS  to  meet  the  requirements  of  the  Army. 
Requirements  included  a  new  central  tire  inflation  system,  a  new  transmission 
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system,  and  unique  axles  to  meet  the  high-density  weight  requirements.  In  view 
of  die  considerable  changes  to  existing  technology  and  major  integration  effort 
required  for  PLS,  a  PRR  should  have  been  performed  before  beginning  LRIP. 

The  Navy  MK-50  program  did  not  perform  annual  incremental  PRRs  during 
engineering  and  manufacturing  development  as  required  by  Military 
.  Standard  1525-B.  The  Military  Standard  requires  the  incremental  review 
process  so  that  the  earlier  PRRs  can  focus  on  gross-level  concerns,  such  as 
identifying  high-risk  and  low-yield  manufacturing  processes  and  producibility  of 
the  proposed  design,  while  the  later  reviews  are  more  refined  and  deal  with 
concerns  such  as  production  planning,  facilities  allocation,  and  fabrication  of 
tools  and  test  equipment. 

Review  Coverage.  As  documented  in  Inspector  General  Report 
No.  93-039,  the  Navy  did  not  perform  PRRs  for  the  overall  EA-6B 
remanufacture  program.  The  only  PRRs  planned  to  be  accomplished  before  the 
LRIP  decision  were  single  reviews  for  two  components  of  the  advanced 
capability  onboard  system,  the  Receiver  Processor  Group  and  the  ALQ-149 
Communications  Jammer.  The  receiver  processor  review  was  scheduled  for 
July  1992,  and  the  ALQ-149  review  was  done  in  August  1990.  In  addition  to 
the  advanced  capability  onboard  system,  the  overall  remanufactured  aircraft  also 
included  the  Vehicle  Enhancement  Program  (maneuverability  enhancements  and 
engine  upgrades)  and  the  Avionics  Improvement  Program  (installation  and 
integration  of  advanced  capability  system  and  vehicle  enhancements).  Those 
additional  parts  of  the  overall  remanufacture  program  were  not  planned  to  be 
subjected  to  PRR  before  LRIP. 

Engineering  Development  Models.  As  discussed,  engineering 
development  models  can  be  used  to  perform  operational  assessments  supporting 
initial  commitment  of  long-lead  procurement  funding  and  entry  into  LRIP.  The 
April  1990  USD(A)  report  to  Congress  proposed  that  DoD  ensure  that 
engineering  development  articles,  used  to  perform  testing  upon  which  initial 
production  decisions  are  made,  be  representative  of  the  production 
configuration.  This  proposal  is  not  in  DoD  Instruction  5000.2.  Delivery 
schedules  for  engineering  development  models  of  systems  were  frequently  not 
met;  therefore,  testing  of  the  models  before  entering  LRIP  was  impaired. 
Additionally,  contractors  did  not  ensure  that  the  models  were  complete  and  met 
contract  specifications.  In  five  of  the  seven  programs,  we  found  that 
contractors  had  not  met  delivery  schedules  established  for  engineering 
development  models  of  systems. 

For  example,  in  Inspector  General,  DoD,  Reports  No.  92-089,  "Cost- 
Effectiveness  Analysis  for  the  Air  Force  C-17  Program,"  May  12,  1992,  and 
No.  91-007,  "Selected  Acquisition  Actions  of  the  C-17  Aircraft,"  November  2, 
1990,  we  found  that  the  C-17  program  had  experienced  major  delays  in  the 
development  of  software  and  hardware  related  to  the  aircraft's  Electronic  Flight 
Control  System  and  the  mission  computer.  These  delays  were  a  primary  cause 
for  the  date  of  the  first  flight  of  the  test  aircraft  to  be  postponed  from  February 
1990  to  September  1991.  In  our  opinion,  additional  time  was  needed  because 
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the  contractor  had  not  recognized  schedule  risk,  had  not  prepared  integration 
test  planning  in  sufficient  detail,  and  lacked  personnel  experienced  in  integrating 
complex  avionics  systems.  When  the  test  aircraft  flew  and  became  a  functional 
test  asset,  the  Air  Force  had  already  issued  contracts  for  three  LRIP  lots,  which 
involved  a  total  of  10  aircraft  at  a  total  ceiling  price  of  $2.9  billion. 

Inspector  General,  DoD,  Report  No.  93-039  disclosed  that  only  one  of  the 
six  models  of  the  Receiver  Processor  Group  (part  of  the  remanufactured 
aircraft's  advanced  capability  onboard  system)  had  been  delivered  as  originally 
scheduled  on  the  EMD  contract.  As  of  July  1992,  the  other  five  engineering 
development  models  were  between  39  and  50  months  behind  the  original 
contract  schedule.  Two  models  were  at  the  contractor's  facilities  and  were 
being  used  for  testing,  but  these  models  had  not  been  accepted  by  the 
Government.  The  delivery  date  for  models  2  through  6  was  set  for  December 
1993. 

In  four  of  the  seven  programs,  we  observed  that  contractors  had  delivered 
engineering  models  of  systems  that  were  incomplete  or  did  not  meet  contract 
specifications. 

For  example,  our  Report  No.  93-039  stated  that  the  only  development  model  of 
the  Receiver  Processor  Group  that  the  Government  had  accepted  was  missing  5 
of  the  21  weapon  replaceable  assemblies  that  were  to  be  included.  Another 
six  assemblies  did  not  meet  configuration  requirements.  As  a  result,  the 
Government  withheld  payment  of  $4  million  on  the  accepted  terminal. 
Additionally,  the  Navy  cited  limited  service  life  remaining  on  the  accepted 
Receiver  Processor  Group  model  as  one  reason  to  shift  reliability  development 
testing  from  the  engineering  and  manufacturing  development  contract  to  the 
LRIP  effort. 

In  the  Air  Force  MILSTAR  Terminal  Program,  models  available  for  testing 
before  the  June  1989  LRIP  decision  were  not  fully  mature  in  design. 
Contractor  schedule  documentation  relating  to  the  EMD  contract  showed  that 
deliveries  of  final  versions  of  development  models  did  not  begin  until  1991.  In 
Inspector  General,  DoD,  Report  No.  93-084,  we  noted  that  the  LRIP  contracts 
were  issued  in  December  1989  when  the  developing  contractor  had  not 
completed  terminal  design.  As  discussed  under  "Unresolved  Design  Problems," 
the  LRIP  contracts  were  awarded  with  "B"  level  specifications  (required 
terminal  capabilities  rather  than  "C"  level  specifications  (design  drawings). 


Entry  Into  Low-Rate  Initial  Production 


The  primary  cause  of  the  deficiencies  identified  was  the  lack  of  a  properly 
planned  and  structured  acquisition  decisionmaking  process  that  provides  for 
essential  oversight  of  program  readiness  to  commence  LRIP. 
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In  our  opinion,  premature  entry  into  LRIP  is  a  systemic  deficiency  that  can  only 
be  corrected  through  a  fundamental  change  in  the  acquisition  oversight  status 
accorded  the  LRIP  decision.  The  present  guidance,  provided  by  DoD 
Instruction  5000.2,  is  intended  to  provide  flexibility  in  structuring  LRIP  within 
acquisition  strategy  to  accommodate  the  unique  aspects  of  individual  programs. 
However,  while  some  flexibility  is  necessary,  basic  systems  engineering 
.management  concepts  such  as  design  maturity,  producibility,  testing,  and 
production  readiness  are  applicable  to  virtually  all  programs.  These  factors 
should  be  thoroughly  reviewed  with  the  overall  program  status  before  entry  into 
LRIP.  Establishment  of  LRIP  as  a  separate  milestone  decision  point  that  can  be 
waived  by  the  milestone  decision  authority  at  Milestone  II,  Development 
Approval,  if  warranted  by  the  nature  of  a  particular  program,  is  required  to 
ensure  program  preparedness  to  proceed  into  LRIP.  While  additional  oversight 
for  LRIP  will  not  substitute  for  sound  program  management,  the  level  of 
oversight  does  directly  effect  the  focus  of  program  management.  In  our 
opinion,  additional  focus  on  entry  into  LRIP  is  warranted. 

Significant  program  changes  and  redirection  often  occurred  after  the  Milestone 
n,  Development  Approval,  decision  that  necessitate  revision  of  acquisition 
strategies,  program  baselines,  and  other  program  parameters.  For  example,  on 
the  seven  programs  reviewed  for  LRIP  readiness,  an  average  of  59  months 
elapsed  between  tide  Milestone  II  decision  and  entry  into  LRIP.  Given  the 
dynamics  of  most  major  programs,  this  period  can  encompass  numerous 
significant  changes  that  have  various  origins  and  levels  of  approval.  All 
programs  were  impacted  to  varying  degrees  by  changes  in  budgets  during  this 
period.  While  Defense  Acquisition  Board  and  Military  Department  program 
reviews  occurred  during  the  59-month  period  in  most  cases,  these  reviews 
varied  significantly  in  focus,  content,  and  required  documentation.  Therefore, 
we  consider  a  clearly  defined  milestone  review  essential. 

Additionally,  DoD  guidance  is  required  on  initial  commitment  of  long-lead 
procurement  funding  and  award  of  additional  LRIP  production  lots  after  entry 
into  LRIP.  Specifically,  exit  criteria  for  these  decision  points  should  be 
required,  not  just  suggested.  Required  exit  criteria  should  include  the 
operational  assessments  recommended  by  USD(A)  in  the  April  1990  report  to 
Congress  before  commitment  of  long-lead  procurement  funding,  completion  of 
critical  design  and  production  readiness  reviews,  accomplishment  of  significant 
levels  of  developmental  testing,  and  at  least  some  operational  testing  before 
entry  into  LRIP.  Additionally,  systems  engineering  and  testing 
accomplishments  should  partially  form  the  basis  for  exit  criteria  for  each  LRIP 
lot  after  entry  into  LRIP.  The  approval  authority  for  these  decisions  should  be 
established  by  the  milestone  decision  authority  upon  entry  into  EMD,  together 
with  the  LRIP  quantities  required  so  as  to  provide  a  basis  for  long-lead  funding 
commitment. 

Regulations  and  Guidance.  Major  Defense  acquisition  programs  are  entering 
LRIP  prematurely  because  of  inadequate  regulation  and  guidance.  Specifically, 
the  DoD  Instruction  5000.2  did  not  provide  adequate  guidance  on: 
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o  Minimum  program  accomplishments  required  before  entering  LRIP  to 
ensure  that  a  stable  design  exists,  test  results  support  proceeding  with  the 
decision,  and  readiness  for  production  has  been  confirmed; 

o  Establishment  of  program-specific  exit  criteria  for  initiation  of  long- 
lead  funding  for  LRIP,  entry  into  LRIP,  and  award  of  subsequent  initial 
production  lots;  and 

o  Milestone  decision  authority  reviews  of  program  status  and 
accomplishments,  including  reaffirmation  of  the  LRIP  quantities  and  acquisition 
strategy  before  entering  LRIP. 

We  believe  that  proper  timing  of  the  LRIP  decision,  which  starts  contractor 
production  lines,  is  critical  to  a  successful  acquisition  strategy.  Once  in 
production,  programs  tend  to  stay  in  production  regardless  of  the  technical  or 
operational  suitability  problems  encountered  with  LRIP  units.  Breaks  in 
production  are  resisted  due  to  the  costs  associated  with  these  breaks,  such  as 
closing  and  later  restarting  production  lines  at  both  prime  and  subcontractor 
levels,  and  the  loss  of  learning  and  experience  that  occurs.  To  give  proper 
emphasis  to  the  LRIP  decision  as  critical  to  the  acquisition  decision  process  and 
to  avoid  the  potential  costly  shutdown  of  production  lines,  we  believe  that  LRIP 
should  be  established  within  DoD  Instruction  5000.2  as  a  separate  milestone 
decision  point. 

In  addition,  DoD  has  issued  a  draft  revision  to  Military  Standard  499A, 
"Systems  Engineering,"  May  1,  1974,  which  does  not  establish  a  direct  link 
between  systems  engineering  requirements  and  provision  of  initial  long-lead 
procurement  funding  and  entry  into  LRIP.  The  draft,  dated  May  6,  1992, 
includes  guidance  on  exit  criteria  for  certain  technical  reviews,  such  as  critical 
design  review,  but  clearly  indicates  that  the  guidelines  are  not  mandatory.  The 
draft  guidance  would  require  revision  to  directly  link  the  systems  engineering 
approach  with  the  LRIP  and  full-rate  production  decision  points  to  be  consistent 
with  the  recommendations  in  this  report.  Based  on  the  systems  engineering 
tasks  defined  in  the  draft  revision  to  Military  Standard  499B,  DoD  has  also 
drafted  a  proposed  Military  Handbook  499-3.  Dated  December  16,  1992,  die 
draft  handbook  proposes  that  functional  configuration  audits  and  a  new 
requirement  called  "Process  Verification"  confirm  readiness  to  enter  LRIP. 
These  assessments  are  conducted  to  demonstrate  that  system  design, 
development,  and  manufacturing  processes  are  sufficiently  mature  to  merit 
initiation  of  LRIP.  The  concepts  in  the  draft  handbook,  if  implemented,  would 
provide  for  significant  reductions  in  program  risk  associated  with  entry  into 
LRIP  and  represent  a  positive  management  initiative  to  correct  the  types  of 
deficiencies  noted  in  this  report. 

Acquisition  Planning.  Major  Defense  acquisition  programs  were  also  entering 
LRIP  prematurely  because  acquisition  plans  for  transitioning  to  production,  set 
when  programs  passed  Milestone  II,  were  developed  when  urgency  to  meet 
threats  justified  higher  levels  of  concurrency  on  the  programs  reviewed, 
including  scheduling  production  decisions  before  design  would  be  complete. 
All  programs  in  our  audit  had  Milestone  II  decisions  before  1988.  As  the 
USD(A)  stated  in  May  1992,  in  the  present  environment,  programs  normally 
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should  not  need  to  incur  the  previously  unacceptable  levels  of  program  risk  in 
transitioning  from  development  to  production.  High  levels  of  concurrency, 
driven  by  demanding  initial  operational  capability  dates,  had  been  justified 
based  on  the  threat;  however,  it  is  not  evident  from  the  programs  reviewed  that 
the  high  concurrency  actually  contributed  to  more  timely  fielding  of  the  weapon 
systems.  * 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

*  As  a  result,  the  Air  Force  was  granted  LRIP 

authority  in  June  1989  and  issued  LRIP  contracts  in  December  1989  when 
terminal  design  was  still  incomplete  (See  "Design  Problems,"  above). 


Impact  of  Present  LRIP  Process 


Increased  program  risk  is  the  primary  impact  of  the  deficiencies  noted  in  the 
LRIP  process.  We  found  that  the  program  risk  associated  with  highly 
concurrent  acquisition  strategies  was  frequently  understated,  in  part  because  risk 
was  not  evaluated  relative  to  the  proposed  guidelines  in  the  April  1990  USD(A) 
report  to  Congress,  which  focuses  on  die  amount  and  results  of  IOT&E 
completed.  The  understated  program  risk  then  impacted  the  amount  of  risk 
management  considered  necessary  by  the  program  officials  and  the  milestone 
decision  authorities. 

When  Military  Departments  prematurely  commit  to  LRIP  on  systems  with  an 
unstable  design,  the  development  and  production  effort  must  be  done  with 
excessive  concurrency  (overlap  between  the  development  and  production 
processes).  This  concurrency  increases  the  potential  for: 

o  Program  cost  increases; 

o  Undetected  design  deficiencies  that  are  difficult  to  fix  in  production 
and  can  impact  achievement  of  required  performance; 

o  Excessive  LRIP  quantities  relative  to  total  planned  program  buys 
driven  by  schedule  delays  and  the  need  to  sustain  production  lines;  and 


*  Predecisional  data  removed. 


22 


Finding  A.  Readiness  for  Low-Rate  Initial  Production 


o  Misuse  of  funds  in  violation  of  law  where  development  and 
procurement  funding  are  not  properly  segregated  and  accounted  for. 

Program  Cost  Increases.  Ideally,  concurrency  can  reduce  the  cost  of  fielding 
major  systems.  However,  undue  concurrency  resulting  from  premature  entry 
into  LRIP  based  on  the  status  of  design,  testing,  and  production  readiness 
invariably  drove  cost  increases  on  the  programs  reviewed.  We  could  not 
quantify  the  precise  dollar  amount  of  cost  increases  on  the  programs  reviewed 
associated  with  higher  levels  of  concurrent  development  and  production  because 
of  the  technical  assumptions  required  on  recognition  and  correction  of  design 
and  producibility  deficiencies  in  less  concurrent  strategies.  Nonetheless,  we 
consider  the  cost  increases  on  these  programs  to  at  least  be  partially  attributable 
to  higher  levels  of  concurrency  risk  introduced  through  premature  LRIP 
decisions. 

Design  Deficiencies.  As  discussed,  six  of  the  seven  programs  reviewed  for 
LRIP  readiness  had  unresolved  design  deficiencies  when  LRIP  started.  The 
introduction  to  DoD  4245.7-M,  "Transition  from  Development  to  Production," 
states  that 


Many  programs  simply  cannot  succeed  in 
production,  despite  the  fact  that  they've  passed  the 
required  milestone  reviews.  These  programs  can't 
succeed  for  technical  reasons,  notwithstanding  what 
is  perceived  as  prior  management  success  related  to 
DoD  acquisition  policy.  A  poorly  designed  product 
cannot  be  tested,  efficiently  produced,  or  deployed. 

In  the  test  program  there  will  be  far  more  failures 
than  should  be  expected.  Manufacturing  problems 
will  overwhelm  production  schedules  and  costs. 

The  best  evidence  of  this  is  the  "hidden  factory 
syndrome"  with  its  needlessly  high  redesign  and 
rework  costs. 

Further,  Inspector  General,  DoD,  Report  No.  93-017,  "Critical  Design  Review 
Process  for  Major  Defense  Acquisition  Programs,"  November  5,  1992,  found 
the  CDR  process  was  not  ensuring  that  design  had  stabilized  before  entering 
production.  Among  the  specific  findings,  design  deficiencies  were  not  being 
resolved  before  stated  completion  of  CDR  with  corrective  action  plans  in  place, 
thus  leading  to  premature  entry  into  production.  The  audit  results  were  based 
on  six  representative  Major  Defense  Acquisition  Programs.  In  the  Inspector 
General  report,  we  recommended  that  CDR  be  completed  before  entering  LRIP 
with  all  design  deficiencies  resolved. 

Excessive  LRIP  Quantities.  Entering  LRIP  before  completion  of  prerequisites 
in  design,  testing,  and  production  preparation  creates  die  potential  for  LRIP 
quantities  that  become  excessive  relative  to  total  planned  program  buys.  The 
potential  for  excessive  LRIP  quantities  occurs  because  production  lines  continue 
as  solutions  are  sought  for  technical  problems.  Finding  B  discusses  this 
problem  and  its  impact  on  the  systems  in  our  audit. 
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Use  of  Funds.  High  levels  of  concurrency  caused  by  premature  LRIP  decisions 
significantly  complicate  accurate  cost  accounting  and  cost  estimating.  While 
properly  structured  contracts  and  sound  contractor  accounting  systems  can 
contribute  significantly  to  proper  cost  accounting,  premature  entry  into  LRIP 
and  the  often-associated  redesign  and  rework  can  complicate  accounting 
decisions.  Therefore,  the  potential  for  misuse  of  appropriated  funds  increases 
when  programs  have  concurrent  development  and  production  efforts  because 
large  amounts  of  Research,  Development,  Test,  and  Evaluation  funding  are 
expended  at  the  same  time  as  procurement  funding. 

The  risk  is  especially  great  when  both  types  of  funding  are  on  the  LRIP 
contract,  as  with  the  Air  Force  MILSTAR  Terminal  and  Titan  IV  Programs.  In 
the  Inspector  General,  DoD,  Report  No.  93-084,  we  found  that  the  Air  Force 
did  not  provide  adequate  oversight  or  control  over  the  expenditures  of  multiple 
appropriations  on  the  LRIP  contracts  for  the  terminals.  As  a  result,  violations 
of  United  States  Code,  title  31,  section  1301  occurred,  which  requires  that 
appropriations  be  applied  only  for  those  purposes  for  which  the  appropriations 
were  made.  Other  fiscal  statutes  could  potentially  be  violated  under  these 
circumstances.  As  a  result  of  the  implementation  of  recommendations  in 
Inspector  General,  DoD,  Audit  Report  No.  92-064,  "Titan  IV  Program," 
March  23,  1992,  the  potential  for  misuse  of  funds  should  be  greatly  reduced 
because  segregation  of  costs  by  appropriations  will  be  required  for  purposes  of 
contractor  payment. 


Conclusions 


To  minimize  the  risk  of  transitioning  from  development  into  LRIP  in  the  present 
reduced-threat  environment,  programs  should  not  normally  transition  until  they 
have  demonstrated: 

o  A  near  absence  of  unresolved  deficiencies  in  developmental  testing; 

o  Successful  completion  of  operational  assessments  on  production- 
representative  units  by  an  independent  Military  Department  test  agency; 

o  Design  maturity,  meaning  no  unresolved  issues  (defined  as  a  lack  of 
agreed-to  fixes)  from  die  technical  reviews,  developmental  testing,  or 
operational  assessments  required  before  LRIP,  which  could  necessitate 
significant  design  changes;  and 

o  Certification  of  the  manufacturing  process  through  successful 
completion  of  PRRs. 

Reaffirmation  of  LRIP  quantities  approved  at  the  Milestone  II  decision  point  is 
required  because  of  the  number  of  program  changes  that  can  occur  between 
Milestone  II  and  the  LRIP  decision.  These  changes  include  budget  reductions, 
test  and  delivery  schedule  slippages,  and  reassessment  of  the  threat 
environment.  Additionally,  to  ensure  that  entry  into  LRIP  is  fully  supported  by 
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program  accomplishments  and  that  planned  numbers  of  LRIP  units  have  been 
properly  defined  in  terms  of  testing  and  production  needs,  we  believe  that  the 
LRIP  decision  should  be  established  as  a  separate  acquisition  milestone. 

In  addition,  a  separate  milestone  review,  which  would  include  confumation  of 
required  initial  production  quantities,  attainment  of  all  required  exit  criteria,  and 
designation  of  approval  authority  for  subsequent  LRIP  contracts,  should  be 
required  to  ensure  programs  are  ready  to  enter  LRIP.  Program  documentation 
listed  in  Appendix  F  should  be  required  for  the  LRIP  milestone  review. 

Our  conclusions  are  supported  by  recent  changes  in  DoD  acquisition  policy. 
The  USD(A)  stated  in  a  May  20,  1992,  acquisition  policy  memorandum  that 
because  of  die  breakup  of  the  Warsaw  Pact  and  the  dissolution  of  the  Soviet 
Union,  the  pressure  of  rapidly  advancing  technology  in  the  hands  of  potential 
enemies  has  significandy  lessened.  The  memorandum  further  states  that  the 
need  to  replace  existing  weapons  systems  to  maintain  a  significant  technological 
advantage  is  no  longer  as  urgent  and  that  program  concurrency  can  be  reduced. 
Fntering  production  prematurely  and  performing  developmental  work  as  part  of 
production  contracts  is  contrary  to  the  intent  of  this  guidance. 

Our  conclusions  are  further  supported  by  the  problems  that  arose  as  a  result  of 
excessive  concurrency  in  programs.  These  problems  are  discussed  in  Finding 
B.  and  documented  in  other  recent  Inspector  General,  DoD,  and  GAO  reports 
(See  Appendixes  C.  and  D.).  Additionally,  our  conclusions  are  consistent  with 
the  USD(A)'s  proposed  (but  only  partially  implemented)  guidelines  for 
explicitly  linking  the  initiation  of  program  milestone  decisions  with 
demonstrated  program  accomplishments  as  discussed  in  the  "Report  on 
Guidelines  for  Determining  the  Degree  of  Risk  Appropriate  for  the 
Development  of  Major  Defense  Acquisition  Systems,  and  Assessing  the  Degree 
of  Risk  Associated  With  Various  Degrees  of  Concurrency;  and  Concurrency  in 
Major  Acquisition  Programs"  issued  April  17,  1990.  The  April  1990  report 
also  proposes  that  concurrency  risk  be  reduced  through  "aggressive"  use  of 
prototyping  and  testing  to  identify  and  remedy  problems  well  before  production 
starts,  citing  the  Packard  Commission  as  the  source  of  this  proposal.  The  LRIP 
recommendations  in  die  USD(A)  report  are  partly  predicated  on  extensive  use  of 
prototyping  before  entry  into  EMD.  We  are  separately  evaluating  prototyping 
in  our  "Audit  of  DoD  Use  of  Prototyping  Acquisition  Strategies  for  Major 
Defense  Acquisition  Programs"  (Project  No.  2AE-0051). 
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Recommendations,  Management  Comments,  and  Audit 
Response 


We  recommend  that  the  Under  Secretary  of  Defense  for  Acquisition: 

1.  Revise  DoD  Instruction  5000.2  concerning  major  Defense  acquisition 
programs  to: 

a.  Establish  a  required  milestone  review  for  entry  into  low-rate 
initial  production,  including  confirmation  of  required  initial  production 
quantities,  attainment  of  all  required  exit  criteria,  and  designation  of 
approval  authority  for  subsequent  low-rate  initial  production  contracts. 
Program  documentation  listed  in  Appendix  F  should  be  required  for  the 
milestone  review. 

b.  Provide  guidance  on  the  specific  minimum  required  program 
accomplishments  for  initially  committing  long-lead  procurement  funding 
for  low-rate  initial  production,  entering  low-rate  initial  production,  and 
awarding  subsequent  low-rate  initial  production  lots.  As  a  minimum, 
operational  assessment,  design  and  production  readiness  review,  and 
operational  testing  prerequisites  should  be  established. 

c.  Require  that  program-specific  exit  criteria  be  established  for 
initial  long-lead  procurement  funding,  entry  into  low-rate  initial 
production,  and  subsequent  initial  production  lots  at  the  Milestone  II 
decision  point  and  incorporated  as  events  in  development  contracts. 

d.  Direct  that  engineering  and  manufacturing  development 
contracts  include  requirements  for  production-representative  engineering 
development  models  for  purposes  of  performing  operational  assessments 
before  initially  committing  long-lead  procurement  funding,  unless 
specifically  waived  by  the  milestone  decision  authority  at  Milestone  n, 
Development  Approval. 

2.  Require  Military  Standard  499A  to  include  a  direct  link  between 
systems  engineering  requirements  and  low-rate  initial  and  full-rate 
production  decisions. 

Management  Comments.  The  Director,  Acquisition  Program  Integration, 
responded  for  the  Under  Secretary  of  Defense  for  Acquisition.  The  Director 
stated  that,  in  general,  he  was  supportive  of  the  finding  and  recommendations, 
but  in  his  view  most  of  the  problems  noted  in  our  report  were  due  to  poor 
compliance  with  current  LRIP  policy  rather  than  the  policy  itself.  The  Director 
concurred  with  Recommendations  A.l.b.  and  A. 2.  and  partially  concurred  with 
Recommendation  A.l.c.  The  Director  nonconcurred  with  Recommendations 
A.l.a.  and  A.l.d. 

Regarding  Recommendation  A.l.a.,  the  Director  stated  that  the  current  policy 
for  optional  LRIP  milestone  reviews  for  Acquisition  Category  I  programs  was 
appropriate  and  that  a  required  milestone  review  before  entry  into  LRIP  is  not 
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needed.  The  Director  did  not  believe  that  the  LRIP  Milestone  IIIA,  provided 
for  in  DoD  Instruction  5000.2  before  February  1991,  had  been  effective  and, 
therefore,  did  not  believe  reintroduction  of  an  LRIP  milestone  would  be  a 
corrective  action.  The  Director  also  believed  that  adding  a  new  milestone  may 
cause  confusion  for  non-major  programs  that  really  need  no  LRIP  review. 

Regarding  Recommendation  A.l.b.,  the  Director  agreed  to  revise  DoD 
Instruction  5000.2  to  provide  increased  guidance  for  major  Defense  acquisition 
programs  on  LRIP  accomplishments  regarding  operational  assessment,  design, 
and  production  readiness  reviews,  and  operational  testing  prerequisites. 

Regarding  Recommendation  A.l.c.,  the  Director  concurred  that,  for  major 
Defense  acquisition  programs,  DoD  Instruction  5000.2  should  be  revised  to  add 
program-specific  exit  criteria  for  LRIP  but  believed  that  the  inclusion  of  the 
criteria  in  development  contracts  should  be  at  the  discretion  of  the  USD(A). 

Regarding  Recommendation  A.l.d.,  the  Director  nonconcurred  with  revising 
DoD  Instruction  5000.2  to  require  engineering  and  manufacturing  development 
contracts  for  major  Defense  acquisition  programs  to  require  production 
representative  engineering  and  development  models  for  LRIP.  The  Director 
stated  that  this  requirement  is  too  stringent  and  that  current  Milestone  II  policy 
already  states  that  LRIP  units  are  "production  configured  or  representative. " 

Regarding  Recommendation  A.2.,  the  Director  agreed  to  revise  Military 
Standard  499A  to  include  a  direct  link  between  systems  engineering 
requirements  and  low-rate  initial  and  foil-rate  production  decisions. 

The  foil  text  of  the  Director's  comments  is  in  Part  IV. 

Audit  Response.  Comments  by  the  Director,  Acquisition  Program  Integration, 
are  partially  responsive. 

o  Concerning  the  response  to  Recommendation  A.l.a.,  we  disagree 
with  management's  assertion  that  a  required  milestone  review  for  entry  into 
LRIP  would  not  help  avoid  the  types  of  problems  noted  during  our  review. 
While  poor  compliance  with  existing  LRIP  policy  was  certainly  a  causative 
factor  to  the  problems  noted,  we  strongly  believe  that  an  equal  or  greater 
causative  factor  was  the  lack  of  policy  guidance  on  the  program-specific 
accomplishments  necessary  to  support  transition  to  LRIP.  Six  of  the 
eight  programs  in  our  review  had  LRIP  milestone  reviews  before  February 
1991,  when  the  September  1987  version  of  DoD  Instruction  5000.2  provided 
for  a  "Milestone  IIIA"  review  before  beginning  LRIP.  However,  there  were 
fundamental  differences  between  the  guidance  for  milestone  reviews  in  the 
earlier  version  of  the  Instruction  and  the  guidance  in  the  February  1991  version. 
The  current  version  of  the  Instruction,  which  does  not  provide  for  a  "Milestone 
IIIA"  review,  gives  detailed  guidance  on  the  objectives  and  decision  criteria  that 
apply  to  each  of  the  currently  established  acquisition  milestones.  The  current 
version  also  provides  excellent  guidance  on  the  minimum  required 
accomplishments  of  each  acquisition  phase  leading  to  a  milestone  and  provides 
for  the  establishment  of  program-specific  exit  criteria.  These  exit  criteria 
provide  event-based  justification  for  making  acquisition  decisions.  We  believe 
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that  the  lack  of  specific  guidance  in  the  earlier  version  of  the  Instruction  made  it 
easier  to  change  or  ignore  established  acquisition  strategies  and  caused  the 
ineffective  LRIP  milestone  reviews. 

A  properly  planned  and  structured  acquisition  decision  process  that  provides  for 
essential  oversight  of  program  readiness  to  enter  LRIP  is  imperative  to  minimize 
.  the  types  of  problems  noted  in  our  report,  especially  with  regard  to  the 
adequacy  of  risk  reduction  efforts  before  production  committments.  The 
oversight  needed  for  the  LRIP  decision  can  best  be  achieved  by  modifying  the 
Instruction  to  require  a  milestone  review  before  entry  into  LRIP  and  by  defining 
the  objectives,  decision  criteria,  and  minimum  required  accomplishments  that 
should  apply  to  that  milestone  similar  to  the  guidance  provided  for  existing 
milestones.  In  addition,  we  disagree  with  management's  assertion  that  a 
required  milestone  review  for  LRIP  could  cause  great  confusion  for  non-major 
programs  that  really  need  no  LRIP  review.  If  there  was  no  valid  need  for  an 
LRIP  review  for  a  non-major  program,  the  milestone  decision  authority  could 
elect  to  grant  a  waiver  to  the  requirement.  Waivers  should  be  granted  only  after 
careful  consideration,  however,  even  when  commercial  products  are  involved. 
LRIP  of  any  product  should  not  begin  until  test  and  evaluation  determine  that 
the  design  has  the  potential  to  meet  military  requirements.  We  ask  management 
to  reconsider  its  position  on  establishment  of  a  required  milestone  review  for 
entry  into  low-rate  initial  production. 

o  Concerning  the  response  to  Recommendation  A.l.b.,  management 
agrees  that  DoD  Instruction  5000.2  should  be  revised  to  provide  increased 
guidance  on  required  program  accomplishments  before  entering  LRIP  but 
provides  no  details  on  what  revisions  will  be  made  and  when  they  will  be 
implemented.  We  ask  management  to  define  what  specific  minimum  program 
accomplishments  the  planned  revision  to  the  Instruction  will  include  relating  to 
initial  commitment  of  long-lead  procurement  funding  for  LRIP,  entering  LRIP, 
and  awarding  subsequent  LRIP  lots.  Additionally,  we  request  management  to 
provide  an  estimated  date  for  the  issuance  of  the  planned  revisions. 

o  Concerning  the  response  to  Recommendation  A.l.c.,  management 
agrees  that  DoD  Instruction  5000.2  should  be  revised  to  add  program-specific 
exit  criteria  for  LRIP  but  is  unclear  as  to  which  LRIP-related  decision  points 
will  require  exit  criteria  under  the  planned  revision  and  when  the  revision  will 
be  made.  Our  recommendation  identified  three  LRIP-related  decisions: 
committing  funding  for  initial  long-lead  procurement,  entering  LRIP,  and 
awarding  subsequent  LRIP  lots.  We  believe  that  program-specific  exit  criteria 
are  needed  for  each  decision  as  a  risk  management  tool  to  ensure  that  funds  are 
not  committed  until  specific  events  or  accomplishments  have  occurred. 
Additionally,  we  believe  management's  position  that  inclusion  of  exit  criteria  in 
development  contracts  should  be  done  at  the  discretion  of  the  USD(A)  is  not 
supported  by  policy  currently  in  DoD  Instruction  5000.2.  The  Instruction 
requires  that  contracting  organizations  must  support  the  acquisition  strategy  by 
imposing  links  between  contract  events  and  demonstrated  accomplishments  in 
development  and  initial  production  and  the  milestone  decisions.  The  events  in 
the  contracts  must  also  support  the  exit  criteria  for  the  phase.  Based  on  the 
guidance  in  the  Instruction,  we  believe  exit  criteria  should  be  incorporated  as 
required  accomplishments  in  development  contracts  unless  waived  by  the 
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USD(A).  We  ask  management  to  reconsider  its  position  on  inclusion  of  exit 
criteria  events  in  development  contracts  and  to  provide  estimated  dates  for  all 
planned  corrective  actions. 

o  Concerning  the  response  to  Recommendation  A.l.d.,  we  agree  with 
management's  assertion  that  the  requirement  for  "production  representative 
models"  for  purposes  of  performing  operational  assessments  before  initially 
committing  long-lead  procurement  funding  may  be  unnecessarily  stringent. 
However,  we  continue  to  believe  that  development  contracts  should  include  a 
requirement  to  provide,  before  commitment  of  long-lead  procurement  funding, 
engineering  and  development  models  that  approximate  the  planned  production 
configuration  for  performing  operational  assessment.  We  agree  that  the 
definition  of  an  engineering  and  development  model  for  this  purpose  requires 
clarification  and  suggest  Military  Standard  499A  contain  a  description  of  what 
constitutes  an  adequate  engineering  development  model  for  operational 
assessment  purposes.  Our  position  is  based  on  the  DoD  Instruction  5000.2 
requirement  that  contracting  organizations  must  support  the  acquisition  strategy 
by  imposing  links  between  contract  events  and  demonstrated  accomplishments  in 
development.  This  requirement  could  be  waived  by  the  Milestone  decision 
authority  at  the  Milestone  II  decision  point,  for  example  if  operational 
assessment  of  prototype  hardware  during  Phase  II,  Demonstration  and 
Validation,  has  disclosed  no  areas  of  significant  (moderate  or  higher)  risk  or 
programmatic  voids.  We  request  management  reconsider  its  position  on  the 
provision  of  engineering  and  development  models  for  operational  assessment 
before  commitment  of  long-lead  procurement  funding. 

o  Concerning  the  response  to  Recommendation  A.2.,  management's 
agreement  to  modify  Military  Standard  499A  is  in  full  concurrence.  We  request 
management  provide  an  estimated  date  for  completion  of  the  planned  revision. 


Other  Management  Comments  and  Audit  Response  to  the 
Comments 


The  Army  Director  for  Combat  Services  Support  disagreed  with  information 
presented  in  the  finding  concerning  the  PLS  program.  He  stated  that  the  finding 
did  not  address  considerations  that  led  the  Army  to  award  the  LRIP  contract 
when  competing  contractors  had  not  met  established  exit  criteria  for  the  award. 
These  considerations  included  a  congressionally  mandated  deadline  to  award  a 
production  contract,  historical  growth  of  reliability  in  other  truck  programs,  and 
the  effect  of  a  delay  in  contract  award  on  competitive  contract  price.  The 
Director  also  questioned  our  reference  for  stating  that  the  PLS  program  had  not 
adhered  to  the  established  acquisition  strategy  and  internal  controls.  The 
Director  additionally  stated  that  the  primary  reason  the  PLS  program  did  not 
have  a  PRR  before  beginning  LRIP  was  that  the  program  had  evaluated 
production  readiness  on  competing  PLS  contractors  during  the  preceding 
2  years. 
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We  believe  the  primary  consideration  in  die  decision  to  begin  LRIP  on  PLS 
should  have  been  the  significant  difference  between  the  1,600  Mean  Miles 
Between  Hardware  Mission  Failures  established  as  an  exit  criteria  for  LRIP  and 
the  900  miles  (56  percent  of  the  exit  criteria)  that  the  contractors  were  able  to 
accomplish.  In  a  July  9,  1990,  memorandum  provided  for  the  Conventional 
Systems  Committee  Milestone  IIIA  (LRIP  decision)  for  PLS,  the  PLS  action 
officer  in  the  Office  of  the  Assistant  Secretary  of  Defense  for  Production  and 
Logistics  stated  that  test  results  for  the  PLS  were  disappointingly  low  and 
indicated  the  program  was  not  ready  to  enter  LRIP.  Concerning  the  acquisition 
strategy  and  internal  controls,  we  believe,  that  by  disregarding  the  established 
exit  criteria,  the  Army  deviated  from  the  acquisition  strategy  and  did  not  follow 
internal  controls  established  to  ensure  that  production  began  only  after  the 
program  had  met  reliability  thresholds. 

In  response  to  the  Director's  comments  that  a  PRR  was  not  necessary  before 
LRIP  because  of  previous  evaluations  of  competing  contractors,  we  found  that 
the  PLS  project  engineer  in  the  Office  of  the  Assistant  Secretary  of  Defense 
(Production  and  Logistics)  had  stated,  in  a  July  9,  1990,  memorandum  provided 
for  the  Conventional  Systems  Committee  Milestone  IILA  (LRIP  decision)  for 
PLS,  that  the  Army  had  not  followed  recommendations  to  conduct  formal  PRRs 
for  the  program.  The  recommendations  for  formal  PRRs  resulted  from  an  April 
1988  DoD  Product  Engineering  Services  Office  assessment  of  the  PLS 
acquisition  strategy.  According  to  the  assessment,  formal  PRRs  were  necessary 
due  to  the  extensive  modifications  and  redesign  efforts  that  would  be  required 
for  competing  contractors  to  develop  a  new  class  of  vehicles  that  could  meet 
Army  specification  requirements. 

Based  on  the  Army  response,  we  have  changed  the  PLS  information  reported  in 
the  finding  by  clarifying  that  the  period  during  which  PRRs  were  not  conducted 
for  the  program  was  up  to  and  including  die  date  of  entry  into  LRIP.  In 
addition,  we  have  deleted  statements  citing  the  nondevelopmental  nature  of  the 
program  as  the  reason  that  PRRs  were  not  performed  before  LRIP. 

The  full  text  of  the  Army  comments  is  in  Part  IV. 

The  Deputy  Assistant  Secretary  of  the  Air  Force  for  Acquisition  disagreed  with 
information  provided  in  the  finding  based  on  audit  results  documented  in 
Report  No.  93-084,  "Air  Force  Strategic  and  Tactical  Relay  Satellite  Terminal 
Program,"  April  13,  1993.  The  Deputy  Assistant  Secretary  disagreed  with  our 
assertion  that  the  Air  Force  MILSTAR  Terminal  Program  went  to  LRIP 
prematurely,  stating  that  the  program  met  criteria  for  start  of  initial  production 
provided  in  the  Program  Management  Directive  for  the  program.  The  Director 
further  stated  that  the  issues  presented  in  Report  No.  93-084  were  still  awaiting 
resolution  at  the  time  our  draft  LRIP  report  was  issued.  The  Assistant  Secretary 
requested  that  we  wait  for  the  resolution  of  these  issues  before  issuing  the  final 
report. 
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We  continue  to  believe  that  the  Air  Force  MILSTAR  Terminal  Program  entered 
LRIP  without  completing  prerequisites  in  design,  testing,  and  production. 
While  the  program  may  have  met  the  specific  criteria  provided  by  the  Program 
Management  Directive  for  start  of  initial  production,  much  research  and 
development  work  still  remained.  In  Report  No.  93-084,  we  noted  that  the 

terminal  program  approached  the  LRIP  decision  * 

* 

* 

* 

*  In  addition  to  the  design, 

testing,  and  production  readiness  shortfalls  mentioned  in  the  finding,  our  review 
determined  that  the  product  configuration  established  at  the  beginning  of  LRIP 
did  not  include  hardware  or  software  changes  that  might  be  required  to  meet  the 
Maintainability  Demonstration  required  under  the  EMD  contract. 

We  completed  resolution  of  outstanding  issues  on  Report  No.  93-084  on 
August  30,  1993.  With  regard  to  the  finding,  these  issues  concerned 
recommendations  for  the  Air  Force  to  identify  the  extent  of  any  violations  of 
fiscal  statues  that  had  occurred  pertaining  to  MILSTAR  terminal  funding  and  to 
establish  procedures  for  proper  control  of  expenditures  by  appropriation.  The 
Air  Force  has  completed  a  review  of  progress  payment  liquidation  against 
deliveries  and  internal  control  procedures  related  to  progress  payments  and 
found  no  basis  for  believing  that  violations  had  occurred.  In  addition,  OSD 
plans  to  require  that  contractor's  requests  for  progress  payments  must  identify 
amounts  by  contract  line  items  or  subline  items  to  the  disbursing  officer.  When 
these  new  guidelines  become  part  of  governing  directives,  the  Air  Force  has 
agreed  to  apply  them  to  the  MILSTAR  terminal  progress  payments. 

Based  on  the  Air  Force  response,  we  have  made  the  following  changes  to  the 
MILSTAR  terminal  information  reported  in  the  finding: 

o  Under  "Design  Problems,"  we  changed  the  description  of  the  52  open 
product  quality  deficiencies  to  include  the  description  provided  in  Report 
No.  93-084:  "This  ongoing  work  included  the  correction  of  52  open  Category 
II  Product  Quality  Deficiency  Reports."  "Category  II  deficiencies  are  defined 
by  Air  Force  Technical  Order  00-35D-54  as  deficiencies  attributable  to  errors  in 
workmanship  or  nonconformance  to  specifications,  drawing  standards,  or  their 
technical  requirements  that  can  result  in  hazardous  or  unsafe  conditions  for 
individuals  using,  maintaining  or  depending  on  the  product  or  prevent 
performance  of  the  product's  tactical  or  strategic  function." 

o  Under  "Review  Performance,"  we  deleted  MILSTAR  from  the 
examples  of  programs  that  had  not  performed  required  annual  incremental 
PRRs. 

The  full  text  of  the  Air  Force  comments  is  in  Part  IV. 


*  Predecisional  data  removed. 


31 


Finding  B.  Low-Rate  Initial  Production 
Quantities  and  Commitments 

The  LRIP  acquisition  strategies  for  major  Defense  acquisition  programs 
did  not  effectively  limit  production  quantities  and  control  program  risk 
before  Milestone  III,  Production  Approval.  Deficiencies  in  planning  and 
executing  die  transition  from  LRIP  to  full-rate  production  were 
identified.  LRIP  acquisition  strategies  were  not  effective  because 
production  decisions  were  scheduled  before  system  design  had  stabilized. 
LRIP  procurement  contracts  were  awarded  while  effort  to  stabilize  the 
design  and  production  process  took  place.  While  the  establishment  of 
exit  criteria  for  planned  production  decisions  was  required  starting  in 
1991,  exit  criteria  were  not  effectively  used  to  control  low-rate  initial 
production  commitments.  As  a  result,  the  Government  incurred  the 
excessive  program  risk  of  over-commitment  to  production  of  systems 
that  have  not  proven  their  production  readiness  or  technical  and 
operational  suitability  and  that  may  have  serious  deficiencies  that  may  be 
difficult  and  costly  to  fix  in  production. 


Background 


The  Milestone  III  Production  Approval  represents  a  commitment  to  build, 
deploy,  and  support  a  system.  It  is  the  last  major  acquisition  decision  before 
systems  move  into  full-rate  production  and  deployment  to  operating  forces. 
DoD  Instruction  5000.2  states  that  the  decision  to  proceed  to  full-rate 
production  should  not  be  made  until  test  results  and  low-rate  initial  production 
experience  provide  reasonable  assurance  that  the  design  is  stable,  operationally 
acceptable,  logistically  supportable,  and  capable  of  being  produced  efficiently. 


Planning  for  Low-Rate  Production 


Lead  time  for  production  of  LRIP  units  was  not  adequately  considered  in 
structuring  LRIP  programs.  As  a  result,  annual  LRIP  contracts  were  awarded 
when  technical  deficiencies  and  program  development  delays  did  not  support 
further  production  commitments.  In  essence,  LRIP  was  used  as  an  extension  of 
engineering  development.  Insufficient  attention  was  focused  on  statutory 
requirements  to  use  LRIP  only  for  the  minimum  essential  quantities  required  for 
operational  test  and  evaluation,  establishment  of  a  production  base,  and  ramp  up 
to  full-rate  production.  Specifically,  minimum  essential  quantities  were 
determined  primarily  by  contractor  or  initial  production  schedules  and  funding 
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availability  rather  than  the  goal  of  minimizing  total  LRIP  quantities.  No 
guidance  was  established  to  determine  minimum  LRIP  quantities  or  required 
analyses  to  support  the  minimum  quantity  and  demonstrate  compliance  with 
United  States  Code,  title  10,  section  2400. 

Annual  Low-Rate  Initial  Production  Contracts.  Four  of  the  seven  programs 
in  our  review  of  Low-Rate  Initial  Production  Quantities  and  Commitments  had 
awarded  (or  planned  to  award)  three  or  more  LRIP  annual  contracts  while  the 
programs  were  experiencing  technical  difficulties  or  program  delays.  Another 
program,  the  Sensor  Fuzed  Weapon,  had  an  acquisition  strategy  that  included 
three  LRIP  awards  before  the  Milestone  III,  Production  Approval.  Exit  criteria 
had  only  been  established  for  the  first  LRIP  award. 

The  low-rate  initial  production  of  the  JTIDS  Class  2  and  Class  2H  terminals,  as 
documented  in  the  GAO  Report  GAO/NSIAD-93-16,  shows  how  technical 
problems  often  have  little  impact  in  determining  whether  LRIP  continues. 
Before  the  award  of  the  second  LRIP  terminal  lot  in  July  1991,  Navy's 
OPTEVFOR  reported  14  limitations  that  prevented  the  test  personnel  from 
determining  the  terminal's  operational  effectiveness  and  suitability.  Although 
the  test  report  showed  that  the  terminals  were  operationally  effective  in  some 
categories,  the  terminals  failed  most  operational  suitability  tests. 

Despite  the  lack  of  adequate  testing  and  poor  test  results,  the  Air  Force  and  the 
Navy  made  a  joint  decision  to  award  Lot  2  production  contracts  to 
two  contractors  in  July  1991  for  53  terminals.  Subsequent  operational  testing  to 
support  a  Lot  3  LRIP  award  was  completed  by  the  Navy's  OPTEVFOR  in 
March  1992.  Although  the  test  report  concluded  that  the  terminals  had  the 
potential  to  be  operationally  effective  and  suitable,  it  stated  that  the  number  of 
significant  deficiencies  was  alarming.  The  OPTEVFOR  report  identified 
five  major  deficiencies  that  needed  to  be  corrected  before  approving  the  system 
for  limited  fleet  introduction  and  53  additional  deficiencies  that  needed  to  be 
corrected  and  verified  by  more  operational  testing.  In  addition,  the  test  report 
also  identified  15  test  limitations  that  prevented  the  resolution  of  critical 
operational  issues.  Despite  these  deficiencies,  the  LRIP  Lot  3  contract  was 
awarded  in  September  1992. 

The  MK-50  program  provides  a  final  example  of  continued  LRIP  awards  despite 
technical  deficiencies  and  program  delays.  The  initial  LRIP  contract  was 
awarded  even  though  the  Technical  Data  Package  was  not  completed,  the 
second  source  was  not  qualified,  and  the  PRR  indicated  many  problems  and 
concerns.  Two  additional  LRIP  contracts  were  awarded  with  the  Technical 
Data  Package  still  incomplete,  the  second  source  still  unqualified,  and  no 
subsequent  required  PRRs  done  to  check  the  original  production  concerns.  A 
fourth  LRIP  contract  was  authorized  by  the  USD(A)  but  was  delayed  due  to  an 
anomaly  report  issued  by  the  Commander  of  Navy  OPTEVFOR. 
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In  our  opinion,  the  continuing  LRIP  awards  occurred  partly  because  programs 
had  entered  UUP  prematurely,  without  meeting  necessary  prerequisites  in 
design,  testing,  and  production  readiness  (see  Finding  A).  Often,  the  programs 
had  entirely  adequate  acquisition  stratejgies  approved,  which  established 
appropriate  prerequisites  for  low-rate  initial  production  decisions,  but  the 
acquisition  strategies  were  either  changed  or  not  adhered  to  before  the  low-rate 
initial  production  decision.  As  a  result,  design  deficiencies  carried  over  into 
LRIP  had  to  be  corrected  while  production  lines  ran.  In  the  worst  case, 
additional  LRIP  contracts  were  awarded  because  of  the  need  to  sustain 
production  lines  even  though  systems  were  not  meeting  established 
requirements.  In  addition,  when  exit  criteria  are  not  established  for  individual 
LRIP  awards,  programs  do  not  have  an  effective  means  of  measuring  progress 
toward  or  justifying  proceeding  with  these  production  decisions.  Appendix  G 
provides  a  summary  of  those  programs  having  awarded  or  planning  to  award 
three  or  more  LRIP  lots. 

Managing  Low-Rate  Initial  Production  Quantities.  We  found  that  additional 
management  attention  is  required  in  establishing  LRIP  quantities  and  holding 
those  quantities  to  the  minimum  needed  assets.  The  number  of  units  generally 
used  in  IOT&E  is  small  compared  to  the  units  produced  under  LRIP.  The 
majority  of  LRIP  effort  falls  within  the  category  of  units  required  to  establish  a 
production  base  and  permit  an  orderly  increase  to  full-rate  production.  This 
later  category  requires  additional  management  attention.  Management  attention 
is  also  needed  when  the  planned  program  quantities  change  significantly,  which 
should  result  in  a  revised  acquisition  strategy  that  can  impact  LRIP  quantities. 
Appendix  H  documents  how  the  LRIP  percentage  of  total  planned  buy  has 
changed  since  Milestone  II. 

Variance  in  Planned  Low-Rate  Initial  Production  Quantities.  In  the 
programs  in  our  audit,  the  LRIP  quantities  planned  at  Milestone  II  varied 
significantly  during  EMD  and  were  different  from  all  present  LRIP  quantities. 
These  changes  support  the  need  for  the  milestone  review  to  confirm  LRIP 
quantities  before  entry  into  LRIP,  as  discussed  in  Finding  A  of  this  report,  and 
the  need  to  separately  identify  and  manage  LRIP  quantity  requirements. 
Specifically,  all  LRIP  units  in  a  program  acquisition  strategy  approved  at  the 
Milestone  II  decision  point  and  above  the  level  certified  by  the  Director, 
Operational  Test  and  Evaluation,  as  necessary  for  IOT&E  should  be  separately 
justified.  The  justification  for  these  LRIP  units  should  be  based  on  the  need  to 
establish  an  initial  production  base  for  the  system  and  permit  an  orderly  increase 
in  the  production  rate  leading  to  full-rate  production  after  completion  of 
IOT&E,  as  specified  in  United  States  Code,  title  10,  section  2400.  The 
justification  for  these  additional  LRIP  quantities  beyond  IOT&E  units  should  be 
reviewed  and  certified  as  part  of  the  production  readiness  review  required 
before  entry  into  LRIP. 

Using  the  production  readiness  review  to  ensure  LRIP  quantities  are  minimized 
provides  for  demonstrable  compliance  with  the  intent  of  United  States  Code, 
title  10,  section  2400  and  is  consistent  with  the  objectives  of  the  reviews  in  DoD 
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Instruction  5000.2.  In  particular,  DoD  Instruction  5000.2  states  that  the 
production  readiness  reviews  should,  among  other  matters,  address  the  cost- 
effectiveness  of  manufacturing  plans,  adequacy  of  production  schedules,  and 
consistency  of  plant  facilities  and  manufacturing  processes  with  production 
quantity  and  rate  requirements. 

The  Army  PLS  program  exemplifies  a  program  with  insufficient  control  over 
LRIP  quantities.  Both  the  LRIP  and  the  full-rate  production  portions  of  the 
program  were  to  be  purchased  under  Contract  DAAE07-90-C-R035.  This 
contract  was  funded  for  a  multi-year  procurement  of  2,691  trucks  and  related 
trailers  during  5-program  years.  The  contract  did  not  put  a  cap  on  the  number 
of  trucks  that  could  be  produced  during  LRIP.  Instead,  the  contract  identified 
an  LRIP  production  rate  of  30  trucks  per  month.  Because  testing  to  establish 
the  truck  configuration  baseline  was  not  completed  as  scheduled  and  identified 
deficiencies  were  not  corrected,  we  determined  that: 

o  LRIP  production  quantities  had  been  expanded,  as  of  December  18, 
1992,  from  592  trucks  to  777  trucks,  an  increase  from  22  percent  to  29  percent 
of  the  planned  2,691  truck  buy;  and 

o  The  Government  had  only  contingently  accepted  the  280  trucks 
delivered,  as  of  November  30,  1992. 

Under  the  terms  of  the  PLS  contract,  LRIP  production  of  30  trucks  per  month 
could  be  continued  as  long  as  production  qualification  testing  is  incomplete  and 
delivery  and  full  Government  acceptance  of  the  489th  truck  has  not  occurred. 
Without  a  cap  on  the  LRIP  quantities,  the  Army  could  complete  production  of 
all  2,691  trucks  without  a  full-rate  production  decision. 

Our  review  of  the  Air  Force  MILSTAR  Terminal  program  showed  how 
approved  LRIP  quantities  can  become  impractical  because  of  program 
reductions.  When  the  USD(A)  issued  a  memorandum  June  18,  1990, 
authorizing  the  Air  Force  to  extend  the  LRIP  program  by  2  years  and  to 
increase  the  original  LRIP  quantities  from  51  to  136  terminals,  a  total  buy  of 
988  terminals  was  planned.  After  the  LRIP  extension,  DoD  made  significant 
reductions  in  the  terminal  program  as  part  of  overall  MILSTAR  program 
restructuring.  As  a  result  of  limited  fund  availability,  only  43  terminals  were 
purchased  on  LRIP  contracts;  the  total  planned  command  post  terminal  quantity 
has  been  reduced  to  *  terminals. 

Despite  the  program  reductions,  the  June  1990  extension  of  LRIP  authority  to 
136  terminals  had  not  been  rescinded.  The  award  of  a  contract  for  the 
remaining  *  terminals  was  being  managed  by  the  Air  Force  as  an  LRIP  "buy 
out,"  or  completion  of  UUP;  in  effect  this  award  represented  a  production 
decision.  As  a  result,  the  Air  Force  did  not  plan  to  complete  IOT&E  as 
required  for  a  full-rate  production  and  deployment  decision  until  after  awarding 
a  contract  for  the  remaining  terminals.  This  situation  was  addressed  in  our 
Audit  Report  No.  93-084.  (See  Appendix  D). 


*  Predecisional  data  removed. 
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LRIP  Quantity  Guidelines.  No  formal  guidelines  existed  concerning 
the  acceptable  quantity  ratio  for  LRIP  versus  the  total  planned  procurement. 
We  found  several  LRIP  "rules  of  thumb"  that  defined  between  10  and 

20  percent  of  the  total  planned  quantity  as  acceptable  for  LRIP.  * 

* 

*  Wedid 

.  not  identify  an  "ideal"  means  to  determine  LRIP  quantities  but  consider 
identification  of  the  factors  in  determining  LRIP  quantities  essential  to  properly 
structuring  LRIP  programs.  While  a  fixed-standard  percentage  or  number  of 
units  is  not  advisable,  we  also  believe  separate  quantification  and  justification  of 
the  number  of  units  required  for  completing  IOT&E,  establishing  a  production 
base,  and  ramping  up  to  full-rate  production  would  improve  the  LRIP  planning 
and  have  the  impact  of  minimizing  LRIP  quantities. 

Effect  of  Proceeding  with  Low-Rate  Initial  Production.  Programs  that 
continue  production  of  systems  without  assurance  that  the  design  is  stable, 
potentially  operationally  acceptable,  and  capable  of  being  produced  efficiently, 
risk  over-commitment  to  production.  Since  the  systems  have  not  proved  their 
technical  and  operational  suitability,  the  Government  then  risks 
three  undesirable  options,  when  there  are  problems  with  the  production  units: 

o  Retrofitting  of  low-rate  initial  production  articles  to  make  them  work 
properly; 

o  Accepting  systems  that  do  not  fully  meet  mission  requirements;  or 

o  Cancelling  systems  after  large  expenditures  for  production. 

Developmental  and  operational  testing  and  technical  reviews  cannot  achieve  the 
goal  of  reducing  risk  by  early  identification  and  correction  of  deficiencies  and 
providing  support  to  program  production  decisions  when  low-rate  initial 
production  decisions  continue  unaffected  by  test  results.  Additionally,  the  tests 
and  reviews  can  be  negatively  impacted  by  efforts  to  proceed  with  low-rate 
initial  production  contracts  when  such  efforts  require  disruption  or  modification 
of  the  content  and  schedules  of  these  activities. 


Conclusions 


We  believe  that  additional  management  attention  is  necessary  to  adequately 
ensure  risks  and  benefits  of  low-rate  initial  production  are  adequately  considered 
before  LRIP  decisions  are  made.  Production  decisions  cannot  normally  be 
justified  when  unresolved  design  issues  and  incomplete  testing,  design,  and 
production  documentation  exist.  Therefore,  we  believe  Service  Acquisition 
Executives  should  be  required  to  request  a  program  waiver  whenever  production 
authority  is  sought  for  a  major  Defense  acquisition  program  before  all 
requirements  for  proceeding  with  the  low-rate  initial  production  decision  in  the 
approved  acquisition  strategy  have  been  met. 


*  Predecisional  data  removed. 
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We  also  believe  that  tighter  controls  are  needed  over  LRIP  quantities  to 
minimize  overcommitment  to  production.  We  have  observed  a  continuing 
pattern  of  programs  being  allowed  to  enter  LRIP  before  completing 
prerequisites  (Finding  A),  then  having  to  issue  multiple  LRIP  awards  to  keep 
production  lilies  running  while  necessary  development  and  design  work 
continue.  We  also  noted  that,  because  of  the  lessening  threat  and  tight 
budgetary  constraints,  programs  have  significantly  cut  back  on  planned  total 
buys.  The  combination  of  extended  LRIP  production  and  reduced  overall  buys 
increases  the  risk  of  overcommitment  to  production.  In  addition  to  separate 
identification,  documentation,  and  approval  of  LRIP  quantities  needed  for 
testing  and  production  considerations,  we  believe  that  cost  and  benefits  of  a 
break  in  production  versus  continued  annual  LRIP  buys  should  be  assessed  by 
the  milestone  decision  authority  before  entry  into  LRIP  to  limit  LRIP  quantities 
to  the  minimum  necessary. 


Recommendations,  Management  Comments,  and  Audit 
Response 


We  recommend  that  the  Under  Secretary  of  Defense  for  Acquisition  revise 
DoD  Instruction  5000.2  concerning  major  Defense  acquisition  programs  to: 

1.  Require  that  the  minimum  low-rate  initial  production  quantities 
necessary  for  initial  operational  test  and  evaluation,  establishment  of  a 
production  base,  and  orderly  increase  to  a  full-rate  production  be 
separately  identified,  documented,  and  approved  by  the  milestone  decision 
authority  at  the  Milestone  II,  Developmental  Approval,  decision  point  and 
reaffirmed  before  entry  into  low-rate  initial  production. 

2.  Review  low-rate  initial  production  quantities  in  the  guidelines  for 
production  readiness  reviews  and  require  certification  as  a  result  of  the 
review  of  the  minimum  low-rate  initial  production  quantity. 

3.  Require  that  the  cost  and  benefits  of  a  break  in  production  versus 
annual  low-rate  initial  production  buys  be  assessed  by  the  milestone 
decision  authority  before  entry  into  low-rate  initial  production  to  limit  low- 
rate  initial  production  quantities  to  the  minimum  necessary  while  providing 
production  units  for  operational  test  and  evaluation. 

4.  Require  Service  Acquisition  Executives  to  request  a  program-specific 
waiver  from  the  Under  Secretary  of  Defense  for  Acquisition  before  award 
of  low-rate  initial  production  contracts  whenever  testing  and  review 
prerequisites  in  DoD  regulations  and  the  approved  acquisition  strategy  are 
deleted,  modified,  or  not  met. 
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Management  Comments.  The  Director,  Acquisition  Program  Integration, 
responded  for  the  Under  Secretary  of  Defense  for  Acquisition.  The  Director 
stated  that,  in  general,  he  was  supportive  of  the  finding  and  recommendations. 
The  Director  concurred  with  Recommendation  B.3.  and  partially  concuped  with 
Recommendation  B.2.  The  Director  nonconcurred  with  Recommendations  B.l. 
and  B.4. 

Full  text  of  the  comments  is  in  Part  IV. 

Regarding  Recommendation  B.l.,  the  Director  stated  that  revision  to  DoD 
Instruction  5000.2  policy  is  not  necessary  because  the  Instruction  already 
requires  definition  of  minimum  LRIP  quantities  for  IOT&E,  initial  production 
base,  and  orderly  increase  to  full-rate  production. 

Regarding  Recommendation  B.2.,  the  Director  agreed  to  include  review  of 
LRIP  quantities  in  the  DoD  Instruction  5000.2  guidelines  for  PRRs  but  did  not 
believe  that  a  requirement  for  certification  of  minimum  LRIP  quantities  as  a 
result  of  the  review  was  necessary. 

Regarding  Recommendation  B.3.,  the  Director  agreed  to  revise  DoD  Instruction 
5000.2  to  require  milestone  decision  authorities  to  assess  the  costs  and  benefits 
of  a  break  in  production  versus  continued  annual  LRIP  buys  before  programs 
enter  LRIP. 

Regarding  Recommendation  B.4.,  the  Director  nonconcurred  with  revising  DoD 
Instruction  5000.2  to  require  Service  Acquisition  Executives  to  request 
program-specific  waivers  before  contract  award  when  LRIP  requirements  are 
not  met.  The  Director  stated  that  the  other  recommendations  in  Findings  A. 
and  B.  will  be  adequate  corrective  action. 

Audit  Response. 

o  Concerning  the  Director,  Acquisition  Program  Integration,  response 
to  Recommendation  B.l.,  we  do  not  agree  that  current  DoD  Instruction  5000.2 
policy  provides  for  separate  identification,  documentation,  and  approval  of  the 
number  of  LRIP  units  required  to  complete  IOT&E,  establish  an  production 
base,  and  ramp  up  to  full-rate  production.  The  Instruction  provides  only  for 
consideration  of  these  three  components  of  LRIP  to  derive  an  overall  LRIP 
amount.  In  addition,  management  comments  do  not  address  our 
recommendation  for  reaffirmation  of  the  LRIP  quantity  before  actual  entry  into 
LRIP.  We  believe  that  the  combination  of  separate  identification, 
documentation,  and  approval  of  the  three  components  of  LRIP  and  reaffirmation 
of  these  quantities  before  entering  LRIP  would  provide  an  essential  control 
technique  for  minimizing  LRIP  quantities.  As  discussed  in  our  report,  many 
changes,  such  as  budget  reductions,  test  and  delivery  schedule  slippages,  and 
reassessment  of  the  threat  can  occur  between  the  Milestone  II  decision  and  entry 
into  LRIP  that  would  cause  a  need  to  reevaluate  the  previously  approved 
quality/rate.  LRIP  reaffirmation  of  quantities  provides  the  opportunity  to 
determine  whether  the  planned  LRIP  quantity  is  still  the  minimum  necessary  to 
achieve  program  production  goals.  Separate  identification  of  the  testing, 
production  base,  and  ramp  up  components  of  the  LRIP  quantity  will  provide 
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documentation  concerning  the  assumptions  behind  the  derivation  of  each 
component  quantity.  Changes  that  occur  in  the  program  can  then  be  assessed, 
as  part  of  the  reaffirmation  of  the  overall  LRIP  quantity,  for  their  impact  on 
each  LRIP  component  quantity.  We  ask  management  to  reconsider  its  position 
on  separate  identification,  documentation,  and  approval  of  LRIP  quantities  and 
to  provide  a  position  on  reaffirmation  of  LRIP  quantities  before  Milestone  II. 

o  Concerning  the  response  to  Recommendation  B.2.,  we  agree  with 
management's  plan  to  include  review  of  LRIP  quantities  into  die  guidelines  for 
PRRs  but  do  not  agree  with  the  assertion  that  certification  of  the  quantities  is 
not  necessary.  We  believe  certification  of  LRIP  quantities  would  provide 
additional  focus  on  minimizing  the  LRIP  quantity,  which  we  consider  a  critical 
production  issue.  As  a  matter  of  clarification,  the  certification  requirement 
could  be  met  by  the  PRR  revalidating  the  LRIP  quantities,  and  for  Major 
Defense  Acquisition  Programs  that  revalidation  could  be  accepted  as  part  of  the 
Production  Readiness  Assessment  provided  to  the  DAB.  We  ask  management 
to  reconsider  its  position  on  certification  of  LRIP  quantities  as  part  of  PRRs  and 
to  provide  an  estimated  date  for  completing  the  revision  of  the  DoD  Instruction 
5000.2. 

o  Concerning  the  response  to  Recommendation  B.3.,  management's 
comments  are  responsive  to  the  intent  of  the  recommendation  but  provided  no 
estimated  date  for  accomplishment  of  the  revision  to  DoD  Instruction  5000.2. 
We  ask  that  management  provide  an  estimated  date  for  completing  revision  of 
the  Instruction. 

o  Concerning  the  response  to  Recommendation  B.4.,  we  disagree  that 
revision  of  DoD  Instruction  5000.2  to  require  Service  Acquisition  Executives  to 
request  waivers  before  contract  award  when  LRIP  requirements  are  not  met  is 
not  necessary  because  of  the  other  recommendations  in  Findings  A.  and  B.  Our 
disagreement  is  based  on  management's  nonconcurence  with  the  following 
provisions  of  the  subject  recommendations:  establishment  of  a  separate 
acquisition  milestone  for  LRIP  and  provision  of  engineering  and  manufacturing 
development  models  for  operational  assessment  before  commitment  of  long-lead 
procurement  funding  for  LRIP.  We  believe  that  the  waiver  of  LRIP 
requirements  should  be  addressed  in  the  Acquisition  Decision  Memorandum 
resulting  from  the  LRIP  milestone  review.  We  ask  that  management  reconsider 
its  position  on  the  requirement  for  waivers  when  required  accomplishments  for 
LRIP  are  not  met. 


Other  Management  Comments  and  Audit  Response  to  the 
Comments 


The  Army  Director  for  Combat  Services  Support  disagreed  with  information 
presented  in  the  finding.  He  stated  that  our  statements  concerning  PLS  LRIP 
quantities  were  misleading  because  they  imply  that  costs  and  risks  for  corrective 
actions  on  the  trucks  fall  on  the  Army. 
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Our  statements  concerning  PLS  LRIP  quantities  were  made  in  the  context  of  an 
example  of  the  need  for  improving  controls  over  LRIP  quantities  to  ensure  that 
these  quantities  are  only  the  minimum  required  for  testing,  establishment  of  a 
production  base,  and  orderly  increase  to  full-rate  production.  As  explained  in 
the  finding,  the  terms  of  the  PLS  contract  made  it  possible  for  the  Army  to 
complete  production  of  the  entire  PLS  buy  without  having  a  Milestone  m, 

.  Production  Approval,  decision. 

Based  on  clarifying  information  in  the  Army  response  we  changed  the  following 
information  in  Finding  B: 

o  We  provided  a  footnote  to  Appendix  G  that  explains  that  individual 
years  of  LRIP  production  were  awarded  as  part  of  a  5-year  multi-year  contract 
that  encompassed  both  LRIP  and  full-rate  production. 

o  We  changed  footnote  1  in  Appendix  H  to  explain  that  the  requirement 
to  specify  LRIP  quantities  became  effective  with  the  Fiscal  Year  1990  Defense 
Authorization  Act. 

The  full  text  of  the  Army  comments  is  in  Part  IV. 

The  Deputy  Assistant  Secretary  of  the  Air  Force  for  Acquisition  disagreed  with 
information  provided  in  the  finding,  which  was  based  on  audit  results 
documented  in  Report  No.  93-084.  The  Deputy  Assistant  Secretary  disagreed 
with  our  assertions  that  LRIP  was  not  used  to  adequately  control  program  risk 
in  transitioning  to  Milestone  III,  Production  Approval,  and  that  the  purchase  of 
the  remaining  terminals  *  should  be  treated  as  a  Milestone  III 

decision  since  it  would  complete  the  terminal  buy. 

We  completed  resolution  of  outstanding  issues  on  Report  No.  93-084,  on 
August  30,  1993.  In  this  earlier  report,  we  recommended  that  USD(A)  require 
that  a  Defense  Acquisition  Board  Milestone  III  Production  Approval  Review, 
supported  by  operational  testing  of  terminals  with  a  MILSTAR  satellite,  be 
conducted  before  completion  of  the  planned  terminal  procurement.  Although 
the  recommendation  was  not  implemented,  the  final  terminal  buy  was  reduced 
from  the  *  terminals  shown  in  Report  No.  93-084  to  38  terminals.  While  we 
continue  to  believe  that  terminal  procurement  is  being  completed  without 
accomplishing  prerequisites  in  design,  testing,  and  production,  we  did  not  seek 
mediation  of  tins  issue  because  we  also  believe  that  the  risk  of  this  acquisition 
has  been  partially  mitigated  by  the  reduction  in  the  terminal  buy. 

The  full  text  of  the  Air  Force  comments  is  in  Part  IV. 


*  Predecisional  data  removed. 
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Appendix  A,  Statutes  Related  to 

Low-Rate  Initial  Production 

United  States  Code, 

title  10,  section  Statute  Excerpt 


138  (a)(2)  In  this  section: 

(A)  The  term  "operational  test  and 
evaluation"  means- 

(i)  the  field  test,  under  realistic 
combat  conditions,  of  any  item  of  (or 
key  component  of)  weapons, 
equipment,  or  munitions  for  the 
purpose  of  determining  the 
effectiveness  and  suitability  of  the 
weapons,  equipment,  or  munitions 
for  use  in  combat  by  typical  military 
users;  and 

(ii)  the  evaluation  of  the  results  of 
such  test. 

(B)  The  term  "major  defense  acquisition 
program"  means  a  Department  of  Defense 
acquisition  program  that  is  a  major 
defense  acquisition  program  for  purposes 
of  section  2430  of  this  title  or  that  is 
designated  as  such  a  program  by  the 
Director  for  purposes  of  this  section. 

2302  (a)(5)  The  term  "major  system"  means  a 

combination  of  elements  that  will  function 
together  to  produce  the  capabilities  required  to 
fulfill  a  mission  need.  The  elements  may 
include  hardware,  equipment,  software,  or  any 
combination  thereof,  but  excludes  construction 
or  other  improvements  to  real  property.  A 
system  may  be  considered  a  major  system  if— 
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United  States  Code, 
title  10,  section 


2302  (continued) 


2399 


Statute  Excerpt 


(A)  the  Department  of  Defense  is 
responsible  for  the  system  and  the  total 
expenditures  for  research,  development, 
test,  and  evaluation  for  the  system  are 
estimated  to  be  more  than  $75,000,000 
(based  on  fiscal  year  1980  constant 
dollars)  or  the  eventual  total  expenditure 
for  procurement  of  more  than 
$300,000,000  based  on  1980  constant 
dollars);  or 

(C)  the  system  is  designated  a  "major 
system"  by  the  head  of  the  agency 
responsible  for  the  system. 

(a)  Condition  for  proceeding  beyond 
low-rate  initial  production. 

(1)  The  Secretary  of  Defense  shall  provide 
that  a  major  defense  acquisition  program 
may  not  proceed  beyond  low-rate  initial 
production  until  initial  operational  test  and 
evaluation  of  the  program  is  completed. 

(2)  In  this  subsection,  the  term  "major 
defense  acquisition  program"  means— 

(A)  a  conventional  weapons  system  that 
is  a  major  system  within  the  meaning  of 
that  term  in  section  2302(5)  of  this  title; 
and 

(B)  is  designed  for  use  in  combat. 

(b)  Operational  test  and  evaluation. 

(1)  Operational  testing  of  a  major  defense 
acquisition  program  may  not  be  conducted 
until  the  Director  of  Operational  Test  and 
Evaluation  of  the  Department  of  Defense 
approves  (in  writing)  the  adequacy  of  the 
plans  (including  the  projected  level  of 
funding)  for  operational  test  and  evaluation 
to  be  conducted  in  connection  with  that 
program. 
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United  States  Code, 
title  10,  section 


2399  (continued) 


2399 


Statute  Excerpt 


(2)  The  Director  shall  analyze  the  results  of 
the  operational  test  and  evaluation  conducted 
for  each  major  defense  acquisition  program. 
At  the  conclusion  of  such  testing,  the 
Director  shall  prepare  a  report  stating  the 
opinion  of  the  Director  as  to— 

(A)  whether  the  test  and  evaluation 
performed  were  adequate;  and 

(B)  whether  the  results  of  such  test  and 
evaluation  confirm  that  the  items  or 
components  actually  tested  are  effective 
and  suitable  for  combat. 

(3)  The  Director  shall  submit  each  report 
under  paragraph  (2)  to  the  Secretary  of 
Defense,  the  Under  Secretary  of  Defense  for 
Acquisition,  and  the  congressional  defense 
committees.  Each  such  report  shall  be 
submitted  to  those  committees  in  precisely 
the  same  form  and  with  precisely  the  same 
content  as  the  report  originally  was 
submitted  to  the  Secretary  and  Under 
Secretary  and  shall  be  accompanied  by  such 
comments  as  the  Secretary  may  wish  to  make 
on  the  report. 

(4)  A  final  decision  within  the  Department  of 
Defense  to  proceed  with  a  major  defense 
acquisition  program  beyond  low-rate  initial 
production  may  not  be  made  until  the 
Director  has  submitted  to  the  Secretary  of 
Defense  the  report  with  respect  to  that 
program  under  paragraph  (2)  and  die 
congressional  defense  committees  have 
received  that  report. 

(5)  In  this  subsection,  the  term  "major 
defense  acquisition  program"  has  the 
meaning  given  that  term  in  section 
138(a)(2)(B)  of  this  title. 
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United  States  Code, 
title  10,  section 


2399  (continued) 


Statute  Excerpt 


(c)  Determination  of  quantity  of  articles  required 
for  operational  testing.  The  quantity  of  articles 
of  a  new  system  that  are  to  be  procured  for 
operational  testing  shall  be  determined  by-- 

(1)  the  Director  of  Operational  Test  and 
Evaluation  of  the  Department  of  Defense,  in 
the  case  of  a  new  system  that  is  a  major 
defense  acquisition  program  (as  defined  in 
section  138(a)(2)(B)  of  this  title);  or 

(2)  the  operational  test  and  evaluation  agency 
of  the  military  department  concerned,  in  the 
case  of  a  new  system  that  is  not  a  major 
defense  acquisition  program. 

(h)  Definitions.  In  this  section: 

(1)  The  term  "operational  test  and 
evaluation"  has  the  meaning  given  that  term 
in  section  138(a)(2)(A)  of  this  title.  For 
purposes  of  subsection  (a),  that  term  does 
not  include  an  operational  assessment  based 
exclusively  on- 

(A)  computer  modeling; 

(B)  simulation;  or 

(C)  an  analysis  of  system  requirements, 
engineering  proposals,  design 
specifications,  or  any  other  information 
contained  in  program  documents. 

(2)  The  term  "congressional  defense 
committees"  means  the  Committees  on 
Armed  Services  and  the  Committee  on 
Appropriations  of  the  Senate  and  House  of 
Representatives.  (Added  Pub. L.  101-189, 
Div.  A,  Title  VIII,  §  802(a)(1),  Nov.  29, 
1989,  103  Stat.  1484.) 
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United  States  Code, 
title  10,  section 


2430 


Statute  Excerpt 


Major  Defense  acquisition  program 
defined. 

In  this  chapter,  the  term  "major  defense 
acquisition  program"  means  a  Department 
of  Defense  acquisition  program  that  is  not  a 
highly  sensitive  classified  program 
(as  determined  by  the  Secretary  of  Defense) 
and- 

(1)  that  is  designated  by  the  Secretary  of 
Defense  as  a  major  defense  acquisition 
program;  or 

(2)  that  is  estimated  by  the  Secretary  of 
Defense  to  require  an  eventual  total 
expenditure  for  research,  development, 
test,  and  evaluation  of  more  than 
$200,000,000  (based  on  fiscal  year  1980 
constant  dollars)  or  an  eventual  total 
expenditure  for  procurement  of  more  than 
$1,000,000,000  (based  on  fiscal  year  1980 
constant  dollars). 


46 


Appendix  B.  Low-Rate  Initial  Production 

Guidance 


DoD  4245.7-M.  "Transition  from  Development  to  Production11:  The 
Manual,  issued  in  September  1985,  provides  assistance  in  structuring  technically 
sound  programs,  assessing  their  risk,  and  identifying  areas  needing  corrective 
action.  The  assistance  is  provided  in  a  series  of  descriptive  templates;  each 
template  discusses  an  area  of  risk  and  then  provides  methods  for  reducing  that 
risk.  The  templates  are  based  on  lessons  learned  from  analysis  of  programs. 
The  DoD  4245.7-M  is  designed  to  provide  a  disciplined  approach  to  the 
acquisition  process  that  will  help  make  decisions  for  ongoing  programs. 


Timelines* 

Program  risk  is  introduced 
when  a  particular  activity 
is  started  late  or  continues 
beyond  the  timeline. 


Points  I  and  II  on  the  table  timeline  are  the  milestones  now  referred  to  in  DoDI  5000.2,  "Defense 
Acquisition  Management  Policies  and  Procedures,"  as  "Concept  Demonstration  Approval"  and 
"Development  Approval,"  respectively.  Point  IIIA  represents  the  decision  to  begin  LRIP,  not 
recognized  as  a  separate  acquisition  milestone  in  DoDI  5000.2;  point  IIIB  represents  the  Milestone  III, 
Production  Approval. 
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Start  and  completion  of  design,  test,  and  production  of  the  activities  listed  in  the 
table  are  given  in  relationship  to  acquisition  milestones.  The  manual  states  that 
program  risk  is  introduced  when  a  particular  activity  is  started  late  or  continues 
beyond  the  timeline.  The  table  provides  that,  for  minimized  program  risk,  28 
of  the  31  activities  (90 percent)  identified  under  "Design,"  "Testing,"  and 
"Production"  be  completed  before  the  LRIP  decision. 

In  addition  to  the  individual  activity  templates  and  activity  time  phasing,  the 
manual  provides  significant  insights  concerning  the  design,  test,  and  production 
efforts  necessary  to  make  a  successful  transition  from  development  to 
production: 

o  Design:  High  risk  of  failure  for  Government  acquisition  programs 
occurs  at  the  outset  of  the  design  process.  While  Some  risk  associated  with  a 
new  technical  concept  may  be  unavoidable,  this  risk  has  been  magnified  by  the 
misunderstanding  of  the  industrial  design  processes  necessary  to  turn  a  concept 
into  a  mature  product.  The  templates  dealing  with  design  address  the  many 
engineering  disciplines  which  must  be  applied  to  reduce  program  risk,  including 
disciplines  that  ensure  the  ability  of  parts  to  endure  stress,  which  have 
historically  been  underemphasized. 

o  Testing:  As  the  system  design  matures,  complex  testing  is  needed  to 
provide  confidence  that  the  system  will  perform  satisfactorily  in  the  operational 
environment.  The  testing-related  templates  are  based  on  test  and  evaluation 
experience  of  major  DoD  programs  and  the  contributions  of  testing  efforts 
toward  reducing  program  risk.  Attention  is  given  to  topics  such  as  integrated 
test  plans;  operational  test  environments;  reliability  development  tests; 
reliability  demonstration  tests;  software  tests;  full-scale  engineering 
development  tests;  initial  operational  test  and  evaluation;  and  applying  the 
process  of  testing,  analyzing  failures,  and  implementing  fixes.  The  guidance  in 
the  templates  addresses  significant  testing  concerns  requiring  management 
attention  to  reduce  the  risk  of  transition  from  development  to  production. 

o  Production:  Solving  the  manufacturing  portion  of  the  acquisition 
equation  is  a  major  factor  in  reducing  the  risk  of  transitioning  to  production. 
The  history  of  military  procurements  includes  many  cases  of  proven  functional 
designs  being  introduced  into  the  manufacturing  process,  only  to  complete  that 
process  as  end  products  that  cannot  support  their  mission  requirements.  The 
templates  provide  guidance  for  early  and  effective  planning  in  areas  that  have 
been  troublesome.  Guidance  covers  subjects  such  as  manufacturing  plans  and 
processes,  quality  control,  subcontractor  control,  tool  planning,  special  test 
equipment,  computer-aided  manufacturing,  and  manufacturing  screening. 

Military  Standard  1521-B.  "Technical  Reviews  and  Audits  for  Systems, 
Equipments,  and  Computer  Software":  The  Military  Standard,  issued  June  4, 
1985,  identifies  requirements  for  technical  reviews  and  audits  which  occur 
throughout  the  acquisition  process.  The  specific  reviews  and  audits  that 
normally  occur  before  the  LRIP  decision  and  their  role  in  providing  feedback 
concerning  program  risk  include: 
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o  Critical  Design  Review:  This  review  is  conducted  for  each 
configuration  item  of  a  system  when  the  detail  design  is  essentially  complete. 
The  purpose  of  this  review  is  to: 

o  Determine  that  the  detail  design  of  the  configuration  item 
reviewed  satisfies  the  performance  and  engineering  specialty  requirements  of  the 
development  specification; 

o  Establish  the  detail  design  compatibility  among  the 
configuration  item  and  other  items  of  equipment,  facilities,  computer  software, 
and  personnel; 

o  Assess  configuration  item  risk  areas  (on  a  technical,  cost,  ancP 
schedule  basis); 

o  Assess  the  results  of  the  producibility  analyses  on  system 

hardware; 

o  Review  the  preliminary  hardware  product  specifications;  and 

o  Determine,  for  software  items,  the  acceptability  of  the  detailed 
design,  performance,  and  test  characteristics  of  the  design  solution  and  the 
adequacy  of  the  operation  and  support  documents. 

o  Test  Readiness  Review:  This  review  for  each  system  software 
configuration  item  determines  whether  the  software  test  procedures  are  complete 
and  ensures  that  the  contractor  is  ready  for  formal  software  testing.  The  review 
also  includes  assessment  of  the  results  of  informal  software  testing  and  updates 
to  the  operational  support  documents.  A  successful  test  readiness  review  is 
predicated  on  the  contracting  agency's  determination  that  the  software  test 
procedures  and  informal  test  results  form  a  basis  for  proceeding  into  formal 
software  testing. 

o  Production  Readiness  Review:  This  review  determines  the  status  of 
completion  of  the  specific  actions  that  must  be  satisfactorily  accomplished 
before  a  production  go-ahead  decision.  The  review  is  accomplished 
incrementally  during  Phase  II,  Engineering  and  Manufacturing  Development. 
Incremental  reviews  are  to  be  at  least  annually  and  before  die  Milestone  HI 
Production  Approval  Review.  In  the  earlier  stages,  the  review  covers  gross- 
level  manufacturing  concerns  such  as  the  need  for  identifying  high-risk  and  low- 
yield  manufacturing  processes  or  materials  or  the  requirement  for  manufacturing 
development  effort  to  satisfy  design  requirements.  The  reviews  become  more 
refined  as  the  design  matures,  dealing  with  concerns  such  as  production 
planning,  facilities  allocation,  incorporation  of  producibility-oriented  changes, 
identification  and  fabrication  of  tools  and  test  equipment,  and  long-lead  item 
acquisition. 

o  Functional  Configuration  Audit:  The  objective  of  this  audit  is  to 
verify  that  the  configuration  item's  actual  performance  complies  with  its 
hardware  development  or  software  requirements  and  interface  requirements 
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specifications.  Test  data  is  reviewed  to  ensure  that  the  computer  hardware  or 
software  performs  as  required.  The  functional  configuration  audit  should  be 
conducted  on  configuration  of  the  item  that  is  representative  of  the  configuration 
to  be  released  for  production  of  the  operational  inventory  quantities. 

o  Physical  Configuration  Audit:  This  audit  is  the  formal  examination  of 
,  the  as-built  version  of  a  configuration  item  against  its  design  documentation  to 
establish  the  product  baseline.  The  audit  includes  detailed  assessment  of 
engineering  drawings,  specifications,  technical  data,  and  tests  used  in 
production  of  hardware  items  and  design  documentation.  For  software  items, 
listings  and  manuals  are  also  examined. 
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Office  of  the  Inspector  General,  Department  of  Defense 


Report  No.  92-089,  "Cost-Effectiveness  Analysis  for  the  Air  Force  C-17 
Program,"  May  12,  1992,  found  that  the  Air  Force  was  overly  committed  to 
production  before  the  Milestone  III  review  since  28  percent  of  the  total  planned 
aircraft  buy  would  already  be  on  contract  when  the  review  was  held.  We 
recommended  that  a  Cost  and  Operational  Effectiveness  Analysis  be  performed 
and  a  special  Defense  Acquisition  Board  program  review  of  the  C-17  Program 
be  conducted  before  the  Lot  V  production  decision.  USD(A)  responded  that  an 
appropriate  amount  of  C-17  review  occurred,  and  an  additional  Defense 
Acquisition  Board  before  Milestone  IIIB  is  not  required.  USD(A)  also 
responded  that  an  additional  Cost  and  Operational  Effectiveness  Analysis 
beyond  current  efforts  is  not  required. 

Report  No.  92-064,  "Titan  IV  Program,"  March  31,  1992,  found  that 
organizations  responsible  for  paying  progress  payments  did  so  without  regard  to 
how  the  costs  were  incurred  relative  to  the  type  of  appropriated  funds.  We 
recommended  that  the  Director  of  Defense  Procurement  issue  policy  guidance 
requiring  that  multi-funded  contracts  include  provisions  to  segregate  progress 
payment  requests  by  appropriation  and  that  the  DoD  Comptroller  review 
accounting  policies  and  procedures  for  all  Military  Departments  to  ensure  that 
adequate  oversight  and  control  of  expenditures  related  to  progress  payments  are 
implemented  and  maintained  as  required  by  DoD  7220.9-M.  Additionally,  we 
recommended  that  the  Program  Executive  Officer  direct  the  Titan  IV 
Procurement  Contracting  Officer  to  request  the  contractor  provide  back-up 
information  to  progress  payment  requests  that  categorizes  costs  incurred  by  type 
of  appropriation. 

Report  No.  92-050,  "Review  of  the  Sensor  Fuzed  Weapon  as  a  Part  of  the 
Audit  of  the  Effectiveness  of  the  Defense  Acquisition  Board  Review  Process," 
February  18,  1992,  found  that  program-specific  exit  criteria  were  not 
established  for  proceeding  to  future  production  decisions.  As  a  result, 
management  did  not  have  an  effective  means  of  measuring  progress  or  justifying 
production  decisions.  We  recommended  the  establishment  of  specific  exit 
criteria  for  future  SFW  production  decisions.  The  USD(A)  concurred  and 
stated  that  program-specific  criteria  were  established  for  die  low-rate  initial 
production  decision,  which  was  made  at  the  Defense  Acquisition  Board  review. 
Proposed  exit  criteria  for  the  full-rate  production  decision  were  discussed  and 
established  for  the  milestone  planned  for  FY  1995. 

Report  No.  91-125,  "Waivers  and  Deviations  to  Contracts  for  the  Airborne 
Self-Protection  Jammer  Program,"  September  30,  1991,  disclosed  that  a 
planned  award  of  46  ASPJ  units  on  an  additional  LRIP  contract  was  not  justified 
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based  on  drastic  reductions  in  the  planned  total  buy.  We  recommended  that  no 
further  ASPJ  contracts  be  awarded  until  completion  of  operational  test  and 
evaluation.  The  USD(A)  generally  concurred  but  determined  that  one  additional 
low-rate  production  award  was  necessary  to  maintain  the  mobilization  base. 

Report  No.  91-007,  "Selected  Acquisition  Actions  on  the  C-17  Aircraft," 

.  November  2,  1990,  found  that  the  Air  Force  System  Program  Office  exercised 
the  FY  1989  contract  option  to  buy  four  production  C-17  aircraft  before 
Douglas  Aircraft  Company  completed  the  required  CDR  of  all  mission 
computer  software.  The  C-17  Program  Director  considered  the  Mission 
Computer  CDR,  done  in  1989,  to  be  adequate  and  complete,  even  though 
Douglas  had  completed  the  detail  design  for  only  about  60  percent  of  all 
software  in  the  mission  computer.  We  recommended  that  the  Air  Force  amend 
the  C-17  full-scale  development  contract  to  require  completion  of  the  software 
design  review  of  the  mission  computer  software  as  a  precondition  to  awarding 
the  Lot  III  production  contract  to  procure  four  aircraft  for  FY  1990.  The  Air 
Force  Deputy  Assistant  Secretary  (Acquisition)  nonconcurred  with  the 
recommendation  and  stated  that  Douglas  met  all  requirements  of  the  CDR  and 
that  Douglas  considered  the  CDR  to  be  completed. 

Report  No.  90-066,  "Hotline  Allegations  Regarding  the  Milestone  HIA 
Production  Decision  for  the  Airborne  Self-Protection  Jammer  Program," 
May  10,  1990,  concluded  that  the  acquisition  plan  for  the  ASPJ  program 
provided  for  die  Navy  to  proceed  with  extensive  production  of  the  ASPJ  units 
before  testing  would  determine  whether  production-representative  ASPJ  units 
would  be  effective  in  their  intended  environment.  The  ASPJ  Acquisition 
Review  Board  inappropriately  influenced  decisionmakers  by  directing  testers  to 
present  favorable  test  results.  The  report  recommended  that  the  Navy 
renegotiate  the  delivery  schedule  for  Lot  I  production  contracts.  The  report  also 
recommended  that  the  USD(A)  defer  the  two  interim  Milestone  HIA  Defense 
Acquisition  Board  reviews  until  operational  test  and  evaluation  confirms  the 
effectiveness  of  the  ASPJ  and  reemphasized  to  acquisition  managers  the 
importance  of  independent  test  and  evaluation  efforts.  Management  generally 
nonconcurred  with  the  recommendations. 

Report  No.  89-104,  "Acquisition  of  the  MK-50  Torpedo  Program,"  August  29, 
1989,  found  that  senior  Navy  and  DoD  decisionmakers  did  not  have  a  full  and 
objective  view  of  the  program's  status  before  transition  into  LRIP.  We  found 
significant  scope  imitations  to  operational  testing  and  reported  test  results  that 
showed  that  the  program  achieved  more  favorable  results  than  were  actually 
demonstrated.  We  recommended  that  the  MK-50  program  not  be  allowed  to  go 
beyond  LRIP  before  it  demonstrated  operational  effectiveness  against  a  target. 

Report  No.  89-075,  "Quick-Reaction  Report  on  Operational  Testing  for  the 
Single  Channel  Ground  and  Airborne  Radio  System  (SINCGARS),"  May  6, 
1989,  found  that  the  Army  was  planning  to  approve  full-rate  production  even 
though  the  program  lacked  the  required  operational  testing  to  proceed  beyond 
LRIP.  We  recommended  that  the  USD(A)  and  the  Director,  Operational  Test 
and  Evaluation,  comply  with  the  United  States  Code,  title  10  regarding 
operational  testing  and  evaluation  being  completed  and  properly  reported  before 
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the  Army  is  allowed  to  award  option  three  (full-rate  production)  of  the  Single 
Channel  Ground  and  Airborne  Radio  System  contract.  The  USD(A)  and 
Director,  Operational  Test  and  Evaluation,  nonconcurred  with  the 
recommendations. 


General  Accounting  Office 


Report  No.  GAO/NSIAD-93-16,  "Military  Communications  Joint  Tactical 
Information  System  Issues,"  November  12,  1992,  revealed  that  the  number  of 
JTIDS'  operating  hours  were  insufficient  to  establish  confidence  in  test  results; 
therefore,  an  adequate  assessment  of  the  terminals  could  not  be  made.  Despite 
these  unsatisfactory  test  and  assessment  results  and  a  recommendation  by  DoD's 
operational  testing  staff  to  delay  production,  the  USD(A)  approved  LRIP  in 
October  1989. 

Report  No.  GAO/NSIAD-93-15,  "Weapons  Acquisition:  A  Rare  Opportunity 
for  Lasting  Change,"  December  1992,  summarized  the  major  acquisition  issues 
addressed  by  GAO  over  the  past  15  years.  The  report  concluded  that  weapon 
systems  have  historically  begun  production  too  quickly  and  were  fielded  with 
major  unknown  or  unresolved  problems.  The  report  made  no  formal 
recommendations  but  concluded  that  pertinent  considerations  for  acquisition 
programs  include  whether  the  acquisition  strategy  is  reasonable,  whether  the 
system  performs  effectively,  and  whether  the  system  is  suitable  for  production 
and  fielding. 

Report  No.  GAO/NSI AD-92- 103,  "Electronic  Warfare:  Established  Criteria 
Not  Met  for  Airborne  Self-Protection  Jammer  Production,"  March  23,  1992, 
revealed  that  after  system  failures  began  during  the  reliability  growth  testing, 
the  Navy  changed  die  criterion  to  exclude  the  system  failures  attributable  to 
software  errors.  With  the  software-induced  failures  excluded,  ASPJ  was  said  to 
have  met  the  reliability  growth  criterion.  As  a  result,  the  Navy  circumvented 
DoD's  testing  standards  and  failed  to  recognize  the  adverse  impacts  of  software 
problems  experienced  with  other  electronic  warfare  systems  similar  to  ASPJ. 
Moreover,  results  of  additional  reliability  growth  testing  conducted  after  the 
Defense  Acquisition  Board  allowed  the  program  to  proceed  shows  that  ASPJ's 
software  problems  are  continuing.  The  GAO  recommended  Congress  oppose 
further  production  contracts  until  operational  tests,  scheduled  for  completion  in 
fiscal  year  1992,  demonstrated  that  ASPJ  will  successfully  perform  its  mission. 
DoD  officials  responded  that,  although  software-induced  failures  were  excluded 
from  the  scoring,  these  failures  had  not  been  ignored  in  the  decision  to  approve 
Lot  II  production.  Officials  said  that  other  tests  conducted  outside  of  the 
reliability  growth  test  program,  including  developmental  flight  tests  and 
contractor  tests  using  modified  software,  had  provided  reasonable  assurance  that 
the  software  problems  had  been  corrected. 

Report  No.  GAO/T-NSIAD-90-43  (OSD  Case  No.  8341-A),  "Weapons 
Systems:  Concurrency  in  the  Acquisition  Process,  Statement  of  Frank  C. 
Conahan,  Assistant  Comptroller  General,  National  Security  and  International 
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Affairs  Division,  before  the  Committee  on  Armed  Services,  United  States 
Senate,"  May  17,  1990,  revealed  that  DoD's  current  acquisition  strategy, 
allowing  production  before  successful  completion  of  critical  tests,  has  resulted 
in  systems  that  did  not  perform  as  intended.  Mr.  Conahan  stated  that  the 
Services  generally  were  not  conducting  or  planning  operational  testing  on 
weapon  systems  until  after  production  started.  Six  weapon  systems  were 
reviewed  and,  in  four  cases,  planned  operational  testing  lagged  behind  the  actual 
or  planned  initial  production  decision  by  1  to  3  years. 

Report  No.  GAO/NSIAD-90-28,  "Military  Satellite  Communications:  Issues 
Associated  With  DoD's  MILSTAR  Terminal  Program,"  May  23,  1990, 
concluded  that  the  decision  authorizing  the  Air  Force  to  proceed  into  LRIP  was 
based  on  less  test  data  than  originally  planned,  thus  increasing  the  risk  that 
design  changes  would  occur.  GAO  recommended  that  the  Air  Force  use 
production-representative  terminals  to  perform  operational  testing  before  the 
Defense  Acquisition  Board  made  the  full-rate  production  decision.  DoD 
agreed,  stating  that  the  issues  identified  would  be  examined  by  the  Defense 
Acquisition  Board  during  its  full-rate  production  review. 

Report  No.  GAO/NSIAD-89-98,  "Navy  Weapons  Testing:  Defense  Policy  on 
Early  Operational  Testing,"  May  8,  1989,  concluded  that  Navy  typically 
proceeded  to  foil-scale  development,  and  often  LRIP,  before  any  Operational 
Test  and  Evaluation  was  completed.  The  GAO  recommended  that  the  Secretary 
of  Defense  reemphasize  the  desirability  of  performing  Operational  Test  and 
Evaluation  as  early  as  possible  in  the  acquisition  cycle,  as  in  DoD's  acquisition 
directives.  The  GAO  also  recommended  the  Secretary  clarify  when  decision¬ 
makers  may  rely  on  operational  assessments  that  may  not  include  the  operational 
testing  of  any  hardware  and  when  actual  operational  testing  and  evaluation  must 
occur. 
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Report  No.  93-084,  "Air  Force  Military  Strategic  and  Tactical  Relay  Satellite 
Terminal  Program,"  April  13,  1993,  revealed  that  the  program  was  not  ready  to 
begin  production  and  deployment  in  March  1993.  Specifically,  the  Air  Force 
had  proceeded  to  a  production  decision  for  *  command  post  terminals, 
expected  to  cost  *  without  completing  the  prerequisites  in  testing, 
design,  and  production.  As  a  result  the  Air  Force  risked  premature  commitment 
to  completing  the  production  of  the  command  post  terminal  and  potential  misuse 
of  funds  in  violation  of  law.  We  recommended  a  Defense  Acquisition  Board 
Milestone  HI  Production  and  Deployment  Review  and  Defense  Acquisition 
Executive  approval  before  award  of  a  contract  for  completion  of  planned 
command  post  terminal  production.  We  also  recommended  that  specific  exit 
criteria,  addressing  terminal  performance,  be  established  for  the  program  to 
enter  full  production.  In  addition  we  found  that  progress  payments  were  made 
from  both  Research  and  Development  and  Procurement  funding  obligated  on  the 
low-rate  initial  production  contracts  without  regard  to  the  amount  of  each  type 
of  work  that  had  actually  been  performed.  We  recommended  implementation  of 
procedures  to  ensure  adequate  internal  controls  over  appropriations  and 
implementation  of  controls  on  LRIP  contracts  to  categorize  costs  properly  and 
ensure  proper  use  of  appropriated  funds.  A  similar  finding  on  mafing  progress 
payments  without  matching  work  performed  to  the  proper  type  of  appropriated 
funds  can  be  found  in  the  synopsis  of  the  Office  of  the  Inspector  General, 
Department  of  Defense,  Report  No.  92-064,  "Titan  IV  Program,"  March  31, 
1992,  included  in  Appendix  D. 

Report  No.  93-039,  "Low-Rate  Initial  Production  of  the  EA-6B  Program," 
December  18,  1992,  revealed  that  the  program  was  to  enter  LRIP  without 
completing  the  prerequisites  in  design,  testing,  and  preparation  for  production 
and  that  the  program  was  being  managed  as  three  separate  segments  rather  than 
as  a  single  major  Defense  acquisition  program.  Also,  we  found  that  testing  to 
determine  the  operational  suitability  and  effectiveness  of  aircraft  manufactured 
during  LRIP  would  not  occur  until  after  the  Milestone  III  Production  Approval. 
In  addition,  procurement  funding,  appropriated  for  the  EA-6B,  was  being  used 
to  fund  development  efforts.  We  recommended  that  the  LRIP  for  the  EA-6B 
aircraft  be  postponed  and  made  contingent  on  specific  exit  criteria  and  that 
program  segments  be  combined  and  managed  as  a  single  major  Defense 
acquisition  program.  We  also  recommended  that  the  results  of  operational  test 
and  evaluation  of  production  representative  aircraft  be  in  the  Milestone  HI 
review.  Additionally,  we  recommended  that  the  program  comply  with 
United  States  Code,  title  31,  sec.  1301  and  sec.  1341,  regarding  the  use  of 
appropriated  funds  for  their  intended  purpose.  The  USD(A)  planned  corrective 
actions  that  met  die  intent  of  all  of  our  recommendations. 


*  Predecisional  data  removed. 
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Quick-Reaction  Memorandum,  "Army  Palletized  Load  System,"  sent  to  the 
Under  Secretary  of  Defense  for  Acquisition  on  December  11,  1992,  stated  that 
the  Army  was  prematurely  proceeding  to  a  Milestone  III,  Production  Approval 
Review.  We  recommended  that  the  Army  complete  additional  operational 
testing  to  verify  that  the  system  can  meet  the  Acquisition  Program  Baseline 
performance  thresholds  before  proceeding  to  full-rate  production.  The  Under 
.  Secretary  of  the  Army  for  Research  and  Development  met  with  the  Director, 
Operational  Test  and  Evaluation's  independent  evaluators  and  recommended, 
based  on  Director,  Operational  Test  and  Evaluation,  input  and  our  memo,  that 
full  production  be  delayed  on  the  PLS  system  until  the  Army  completes 
additional  testing  to  resolve  these  issues. 
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1  The  deficiencies  were  noted  on  this  program  before  funding  was  cancelled. 

2  Incomplete  testing  also  indicates  potential  design  instability,  since  design  changes  are  often  driven  by  test 
results. 
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PLS  MK-501  EA-6B  MILSTAR  ASPJ  C-17  JTIDS 

-  Engineering  Models 
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1  The  deficiencies  were  noted  on  this  program  before  funding  was  cancelled. 
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Requirements1 

Operational  Requirements  Document 

System  Threat  Assessment  Report 

Defense  Intelligence  Agency  Intelligence  Report2 

Joint  Requirements  Oversight  Council  Assessment2 

Integrated  Program  Summary 

Integrated  Program  Assessment 

Program  Life-Cycle  Cost  Estimate 

Acquisition  Program  Baseline  Agreement3 

Manpower  Estimate  Report 

Test  and  Evaluation  Master  Plan 

Developmental  Test  and  Evaluation  Report 

Independent  Cost  Estimate 

Independent  Cost-Estimate  Report 

Cost  and  Operational  Effectiveness  Analysis 

Early  Operational  Assessment  Report  (should  include  results  of  some  early 
initial  operational  test  and  evaluation) 


1  With  exception  of  the  Developmental  Test  and  Evaluation  Report  and  the  Early  Operational 
Assessment,  documentation  requirements  would  be  limited  to  any  updates  required  to  account  for 
changes  occuring  since  Milestone  II  approval. 

2  DoD  Instruction  5000.2  requires  these  documents  for  Acquisition  Category  ID  programs  only. 

3  The  continued  viability  of  performance,  schedule,  and  cost  objectives  and  threshold  provided  in  the 
Baseline  set  at  Milestone  II  should  be  assessed  and  adjusted  as  necessary.  Baseline  schedule  dates 
proceeding  the  Milestone  II,  Production  Approval,  should  include  "date  complete"  for  functional  and 
physical  Configuration  Audits  and  for  final  (before  full-rate)  Production  Readiness  Review;  "Date  of 
Delivery"  for  first  LRIP  Unit;  and  Start/Complete  dates  for  IOT&E. 
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Appendix  G.  Multiple  Annual  Low-Rate  Initial 

Production  Contracts  Awarded  or 
Planned 


1988  1989  1990  1991  1992  1993  1294 


Program 


C-47  Aircraft  XX  XXX 


Joint  Tactical 
Information 
Distribution  System 


MK-50  Torpedo  liH  !;i 


Palletized  Load  System 


Seh|#:i5S||d?Weaipbhi 
System  1 


XXX 

:: 

X*  X*  X*  X * 

lilt  4SX'  F-'X:P  X 


*  Four  individual  production  years  for  PLS  were  awarded  as  part  of  a  five  year  multi-year  contract  that 
encompassed  both  LRIP  (1990-1993)  and  full-rate  production. 
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Appendix  H.  Low-Rate  Initial  Production  as  a 

Percentage  of  Total  Planned 
Production 

Total  Total  LRIP  Percent  LRIP  Percent 

Quantities  Planned  Quantities  Planned  of  the  Planned  of  the  Planned 

at  Milestone  II  as  of  December  1992  Total  Buy  at  Total  Buy  as  of 

Program  Total  Buv  LRIP  Total  Buv  LRIP  Milestone  II  December  1992 

ASPJ  533  01  739  136  0  18 

SFW 

14,000 

1,080 

10,000 

2,022 

8 

20 

C-17 

210 

O1 

120 

28 

0 

23 

PLS 

4,333 

O1 

2,691 

777 

0 

29 

EA-6B 

102 

21 

102 

15 

21 

15 

MILSTAR  988 

51 

* 

* 

5 

1002 

MK-50 

C3 

436 

C3 

851 

N/A 

N/A 

JTIDS 

1,179 

278 

971 

166 

24 

17 

1  The  requirement  to  specify  LRIP  requirement  began  with  the  fiscal  year  1990  Defense  Authorization 
Act. 

2  Total  buy  was  being  executed  under  LRIP  authority . 

3  Total  quantities  are  classified  (C). 

*  Predecisional  data  removed. 
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Appendix  I. 

Recommendation 

Reference 

A.l.a. 

A.l.b. 

A.l.c. 

A.l.d. 

A.2. 

B.l.  and  B.2. 


Summary  of  Potential  Benefits 
Resulting  From  Audit 


Amount  and/or 

Description  of  Benefit  Type  of  Benefit 


Internal  Control.  Will  ensure  that 
LRIP  begins  after  the  milestone 
decision  authority  has  fully  assessed 
the  required  LRIP  quantities  and  the 
attainment  of  required  program 
accomplishments  and  exit  criteria. 

Internal  Control.  Will  provide 
guidance  on  minimum 
accomplishments  programs  must 
meet  before  commiting  resources 
for  LRIP. 

Internal  Control.  Will  ensure  that 
acquisition  decisions  are  based  on 
specific  program  accomplishments. 

Internal  Control.  Will  ensure  that 
operational  assessments  of 
engineering  development  models 
can  be  performed  before  committing 
long-lead  procurement  funding  for 
LRIP. 

Internal  Control.  Will  provide 
direct  linkage  between  systems 
engineering  requirements  and  low- 
rate  initial  and  full-rate  production 
decisions. 

Compliance  with  Public  Law  and 
Internal  Control.  Will  ensure  that 
LRIP  quantities  are  limited  to  the 
minimum  necessary  to  support 
operational  testing  and  production 
considerations. 


Nonmonetary. 


Nonmonetary. 


Nonmonetary. 


Nonmonetary. 


Nonmonetary. 


Undeterminable. 
Amount  will  vary 
with  each  program 
and  benefits  will 
continue  as  new 
programs  define  LRIP 
quantities. 
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Appendix  I.  Summary  of  Potential  Benefits  Resulting  From  Audit 


Recommendation 

Reference 


B.3. 


B.4. 


Amount  and/or 

Description  of  Benefit  Type  of  Benefit 


Internal  Controls.  Will  ensure  that 
cost  and  benefits  of  a  production 
line  break  are  fully  considered  as  an 
alternative  to  continued  LRIP  buys. 


Internal  Controls.  Will  ensure  that 
low-rate  initial  production  contracts 
are  awarded  only  after  all  testing 
and  review  prerequisites  have  been 
met  unless  a  specific  waiver  is 
provided. 


Undeterminable. 
Amount  will  vary 
with  each  program 
and  benefits  will 
continue  as  future 
programs  perform  the 
cost  and  benefits 
analysis. 

Nonmonetary. 
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Appendix  J.  Activities  Visited  or  Contacted 


Office  of  the  Secretary  of  Defense 

Under,  Secretary  of  Defense  for  Acquisition,  Washington,  DC 

Assistant  Secretary  of  Defense  (Program  Analysis  and  Evaluation),  Washington,  DC 

Comptroller  of  the  Department  of  Defense,  Washington,  DC 

Director,  Operational  Test  and  Evaluation,  Washington,  DC 


Department  of  the  Army 

Deputy  Chief  of  Staff  for  Operations  and  Plans,  Washington  DC 
Cost  and  Economic  Analysis  Center,  Alexandria,  VA 

Materiel  Systems  Analysis  Activity,  U.S.  Army  Materiel  Command,  Aberdeen  Proving 
Ground,  MD 

Operational  Test  and  Evaluation  Agency,  Alexandria,  VA 
Program  Executive  Office,  Combat  Support,  Warren,  MI 
Palletized  Load  System  Program  Office,  Warren,  MI 


Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition), 
Washington,  DC 

Naval  Command  Control  and  Ocean  Surveillance  Center,  San  Diego,  CA 
Naval  Operational  Test  and  Evaluation  Force,  Norfolk,  VA 
Navy  Program  Executive  Office,  Surface  Ship  Anti-Submarine  Warfare, 
Washington,  DC 

A-6/EA-6  Program  Office,  Naval  Air  Systems  Command,  Washington,  DC 
Joint  Terminal  Program  Office,  Washington,  DC 

MK-50  Torpedo  Program  Office,  Naval  Sea  Systems  Command,  Washington,  DC 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Acquisition)  Washington,  DC 
Air  Force  Program  Executive  Office,  Space  Programs,  Washington,  DC 
Air  Force  Program  Executive  Office,  Tactical  Systems,  Washington  DC 
Air  Force  Operational  Test  and  Evaluation  Center,  Kirtland  Air  Force  Base, 

New  Mexico 

Military  Strategic  and  Tactical  Relay  Satellite  Program  Office,  Space  Systems  Center, 
Los  Angeles,  CA 
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Appendix  J.  Activities  Visited  or  Contacted 


Department  of  the  Air  Force  (Continued) 

Military  Strategic  and  Tactical  Relay  Terminal  Satellite  Program  Office,  Electronic 
Systems  Center,  Hanscom  Air  Force  Base,  MA 


Other  Defense  Organizations 

Defense  Logistics  Agency,  Cameron  Station,  Alexandria,  VA 

Defense  Contract  Audit  Agency,  Raytheon  Company,  Burlington,  MA 

Defense  Contract  Management  Administrative  Office,  Milwaukee,  WI 

Defense  Plant  Representative  Office,  Grumman  Aerospace  Corporation,  Bethpage,  NY 

Defense  Plant  Representative  Office,  Oshkosh  Truck  Corporation,  Oshkosh,  WI 

Defense  Plant  Representative  Office,  Raytheon  Company,  Burlington,  MA 

Defense  Finance  and  Accounting  Service,  Columbus,  OH 


Non-Defense  Activities 

Grumman  Aerospace  Corporation,  Bethpage,  NY 
Oshkosh  Truck  Corporation,  Oshkosh,  WI 
Raytheon  Company,  Burlington,  MA 

Raytheon  Company,  Equipment  Division,  Satellite  Terminal  Systems  Directorate, 
Marlborough,  MA 
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Appendix  K.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition 
Director,  Operational  Test  and  Evaluation 
Comptroller  of  the  Department  of  Defense 


■  Department  of  the  Army 

Secretary  of  the  Army 

Assistant  Secretary  of  the  Army  (Research,  Development,  and  Acquisition) 
Auditor  General,  Department  of  the  Army 
Deputy  Commanding  General  for  Army  Materiel  Command 
Program  Manager,  Palletized  Loading  System 


Department  of  the  Navy 

Secretary  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management) 

Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition) 

Commander,  Naval  Air  Systems  Command 

Commander,  Naval  Sea  Systems  Command 

Auditor  General,  Naval  Audit  Service 

Program  Manager,  Airborne  Self-Protection  Jammer 

Program  Manager,  A-6/EA-6  Aircraft 

Program  Manager,  MK-50  Torpedo 


Department  of  the  Air  Force 

Secretary  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Acquisition) 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 

Commander,  Air  Force  Materiel  Command 

Auditor  General,  Air  Force  Audit  Agency 

Program  Executive  Officer  for  Space  Programs 

Program  Manager,  C-17  System 

Program  Manager,  Joint  Tactical  Information  Distribution  System 

Program  Manager,  Military  Strategic  and  Tactical  Relay  Satellite  Systems  Terminal 

Program  Manager,  Sensor  Fuzed  Weapon 
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Appendix  K.  Report  Distribution 


Defense  Agencies 

Director,  Defense  Contract  Management  Command 
Director,  Defense  Logistics  Agency 
Commander,  Defense  Contract  Management  Command 
Director,  Defense  Finance  and  Accounting  Service 
Inspector  General,  Defense  Intelligence  Agency 
Inspector  General,  National  Security  Agency 
Director,  Defense  Logistics  Studies  Information  Exchange 


Non-Defense  Federal  Organizations 

Office  of  Management  and  Budget 

U.S.  General  Accounting  Office,  National  Security  and  International  Affairs  Division, 
Technical  Information  Center 

Chairman  and  Ranking  Minority  Member  of  the  Following  Congressional  Committees 
and  Subcommittees: 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 

House  Committee  on  Armed  Services 

House  Committee  on  Governmental  Operations 

House  Subcommittee  on  Legislation  and  National  Security,  Committee  on 
Government  Operations 
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This  page  was  left  out  of  orignial  document 


Part  IV  -  Management  Comments 


Office  of  the  Under  Secretary  of  Defense  for  Acquisition 
Department  of  the  Army 
Department  of  the  Air  Force 


Office  of  the  Under  Secretary  of  Defense  for 
Acquisition  Comments 


ACOlHtlT'ON 


OFFICE  OF  THE  UNDER  SECRt.  .  ARY  OF  DEFENSE 
WASHINGTON,  DC  S030 1-9000 

ii  ik  wn 


MEMORANDUM  FOR  DIRECTOR,  ACQUISITION  MANAGEMENT 

DIRECTORATE,  OFFICE  OF  THE  DOD  INSPECTOR  GENERAL 

SUBJECT:  Draft  DoD  Inspector  General  Audit  Report  on  the  Low- 
Rat*  initial  Production  in  Major  Defense  Acquisition 
Progress  (Project  No.  2AE-0026) 


ThenX  you  for  the  opportunity  to  review  the  draft  audit 
report  concerning  low-rate  initial  production  (LRIP) . 

In  general,  we  are  supportive  of  your  findings  and 
recommendations,  but  would  note  that  in  our  view  most  of  the 
problems  noted  in  your  report  are  due  to  poor  cospllanoe  with 
current  LRIP  policy  --  not  the  policy  itself.  Por  example,  the 
LRIP  problems  in  the  C-17  and  ASPJ  progress  were  not  as  such  a 
shortcoming  in  LRIP  policy  as  they  were  a  failure  to  take  heed  of 
warning  signs.  On  page  31  of  your  draft  report  you  adalt  that 
"Often,  the  programs  had  entirely  adequate  acquisition  strategias 
approved,  which  established  appropriate  prerequisites  for  low- 
rate  production  decisions,  but  the  acquisition  strategies  were 
either  changed  or  not  adhered  to  before  the  low-rate  production 
decision".  Most  of  the  reasons  that  program  plans  are  not 
adhered  to  are  related  to  unforeseen  technical  problems  or 
substantial  reductions  in  planned  quantlties/progran  funding. 

Long  lead  LRIP  items  are  typleally  contracted  for  11-24  months  in 
advance  of  production  and  are  therefore  on  contraot  well  before 
technical,  fiscal,  or  programmatic  problems  surface.  Quite  often 
it  coste  more  to  change  the  contract  than  to  proceed  as  planned. 
We  agree  that  expanding  the  9000.2  guidance  on  LRIP  standards 
should  help  but  we  cannot  see  how  re-introducing  the  Milestone 
IIIA  will  help  much  as  the  eight  Milestone  XXXAs  eited  in  your 
draft  were  apparently  ineffectual. 


Please  be  advised  that  the  name  change  from  USD(A)  to 
USD (AST)  that  you  anticipated  in  your  June  1*93  draft  report  did 
not  occur.  The  USD(A)  remains  the  official  title.  Me  understand 
that  the  Air  Pores  is  also  communicating  directly  with  you 
regarding  factual  errors  in  the  report  regarding  Milstar. 

Our  detailed  responses  to  individual  recommendations  are: 
ElMDIMO  A  -  Readiness  for  U>T> 

Recommendation  recommends  that  DoDI  5000.2  be  revised 
to  include  a  new  milestone  for  mdafb  prior  to  entry  into  lair 
(the  5000.2  policy  prior  to  February  199 1  Included  the  LRIP 


70 


Office  of  the  Under  Secretary  of  Defense  for  Acquisition  Comments 


Milestone  I I IX) .  W.  nonconcur  in  thin  reeoanendetion.  All  tight 
of  the  program*  (PLS,  KX-50 ,  Milstar  Terainels,  ASPJ,  C- 

17,  $pw,  and  JTIDS)  want  through  an  LRIP  Milestone  II1A  and  yat 
you  critielta  aach  program  and  make  thia  recommandation  to  ra- 
introduca  tha  Milaatona  XZIA.  Xt  ia  not  elaar  how  tha 
re Introduction  of  an  LRIP  Milaatona  XXXA  will  ba  corractive 
action  whan  all  caaaa  raportad  did  includa  a  milaatona 
oonaidaration  of  LRIP.  Our  currant  policy  atataa  that  for  ACAT  1 
programs  a  raviaw  may  ba  rtqulrad  prior  to  LRXP.  Wa  raaiat 
adding  a  naw  ailaatona,  ainca  5000.1  policy  appliaa  to  all 
acquisitions,  not  just  MDAPs.  A  larga  majority  (naar  50%)  of 
all  acquisition  programs  ara  not  major*  As  DoD  incraaaingly 
turns  to  mors  commercial  products,  wa  may  eauaa  graat  confusion 
for  non-major  programs  that  raally  naad  no  LRIP  raviaw  (e.g., 
Chevy  Blazers  off  tha  commaroial  production  lina  for  Army  CUCV 
uaa)  •  Wa  faal  that  our  currant  policy  for  optional  LRIP  on  MDAPs 
la  appropriate.  What  ia  needed  is  tha  increased  rigor  you  note 
in  other  recommendations,  nat  Milestone  XIXA. 

Recommendation  l.b,  recommends  that  for  MDAPs ,  S000.2  ba 
revised  to  provide  increased  guidance  on  LRIP  accomplishments 
regarding  operational  assessment,  design,  and  production 
readiness  review,  and  operational  teat  prerequisites.  Wa  ooncur. 

Recommendation  l.c.  recommends  that  for  MDAPs,  5000.2  be 
revised  to  add  program  specific  exit  criteria  for  LRXP.  Wa 
concur  except  that  tha  lncluaion  of  tha  criteria  in  development 
contracts  should  ba  at  discretion  of  USD (A), 

Recommendation  l.d.  recommends  that  for  MDAPs,  5000.2  ba 
revised  to  direct  2M0  contracts  to  includa  production 
representative  models  for  LRIP.  Wa  nonconcur  as  this  Is  too 
stringent,  and  our  currant  Milestone  XI  policy  already  states 
that  LRIP  units  ara  "production  configured  or  representative.1' 

Recommendation  Lx  recommends  a  revision  to  MilStd  499A.  Wa 
concur. 


R.coaa.ndation  l*  recoaaends  revision  of  S000.2  to  roguiro 
alniaua  LRIP  quantities  for  IOTtE,  initial  production  base,  and 
orderly  increase  to  full-rate  production.  We  nonoonour  as  we 
feel  this  le  already  5000.2  policy  (pert  2-15). 

Recoaaendation  2*  reeoaaende  revision  of  5000.2  to  include 
LRXP  quantities  in  PRR  guidelines.  Me  concur  but  fool 
eertifleetion  is  not  necessary. 
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Recommendation  Jj.  recommends  revision  of  5000.2  to  include 
coat/benefit  analysis,  before  entry  into  1RIP,  of  break  in 
production  vereue  UUP.  He  concur. 

Recommendation  ju  recommends  revieion  of  5000.2  to  require 
BAEs  to  requeet  program  specific  vaivere  before  eontrect  sward 
when  LRIP  requirements  ere  not  set.  We  nonconcur  because  the 
other  recommendations  in  Findings  A  end  B  above  will  be  adequate 
corrective  set ion. 


Gens  H.  Porter' 
Director,  Acquisition 
Progrea  Integration 
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Department  of  the  Army  Comments 


DEPARTMENT  OF  THE  ARMY 
OTFICE  OF  THE  assistant  secretary 

WASHINGTON.  DC  20010-0103 


1  0  AUG  1993 

SARD-ZCS 


MEMORANDUM  FOR  Inspector  General,  Departmant  of  Defense  (Auditing) 

SUBJECT:  DoD  Inspector  General  Draft  Report,  Low-Rate  Initial 
Production  in  Major  Defense  Acquisition  Programs  (Project  No,  2AE- 
0026) 


1.  Reference  Draft  Audit  Report,  Low-Rate  Initial  Production  in 
Major  Defense  Acquisition  Prograss,  Project  No.  2AE-0026,  17  June 
1993. 

2.  The  following  inforsation  and  cosnents  are  provided, 

3 .  Page  14,  Unresolved  Deficiencies. 

This  paragraph  describes  the  Palletized  Load  System  (PLS)  entry 
into  LRXP  and  production  award  with  the  less  than  required  1600 
Kean  Miles  Between  Hardware  Mission  Failures  (MMBHMF) .  Not 
mentioned  are  the  following  considerations  that  led  Army  to  this 
decision. 

a.  Normal  RAM  growth  usually  experienced  during 
developmental  testing  was  prevented  by  test  and  time  constraints. 
Army  was  under  a  Congressional ly  mandated  deadline  to  make  a 
production  contract  award. 

b.  The  PLS,  while  somewhat  more  complex  than  a  typical 
truck,  does  not  incorporate  high  technology.  The  failures 
experienced  were  not  unusual  for  a  truck  system. 

c.  Historically,  other  truck  programs  have  experienced  a  low 
initial  RAM  score  and  have  made  dramatic  improvements  through  the 
initial  production  phase  similar  to  that  needed  for  the  PLS. 

d.  Additional  testing  was  planned  for  and  conducted  shortly 
after  the  LRIP  decision  to  assure  the  needed  RAM  growth  was 
attained. 

e.  A  competitive  contract  price  would  have  been  lomt  if 
delays  to  the  contract  award  were  experienced. 

As  it  is,  the  better  decision  was  make.  RAM  requirements  were  met 
and  the  program  maintained  its  overall  schedule  while  still 
benefitting  from  a  coapatltlva  price. 


Final  Report 
Reference 


13 
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Final  Report 


SUBJECT:  DoD  Inspector  General  Draft  Report,  low -Rate  initial 
Production  in  Major  Defense  Acquisition  Programs  (Project  Ho.  2AE- 
0026) 


4.  Page  19,  Review  Performance: 

The  report  states  that  "Ho  Production  Readiness  Reviews  (PRR)  were 
performed  for  PLS  because  PLS  was  considered  a  nondevelopmental 
item  (HDI) ■ .  He  disagree  with  this  statement  on  two  counts: 

a.  The  decision  not  to  have  a  PRR  prior  to  the  Milestone 
IIIA  DAB  decision  was  not  based  on  PLS  being  an  HDI.  Our  primary 
reasoning  for  not  pursuing  a  PRR  was  that  we  had  gone  through  two 
rounds  of  evaluating  production  readiness  on  all  PLS  competitors 
during  the  preceding  two-year  period  (as  part  of  the  prototype  and 
production  source  selections).  Most  of  the  major  systems  entering 
LRIP  have  not  had  any  competitive  source  selection  evaluation 
(including  pre-award  surveys  for  a  long  period  because  the  current 
process  of  treating  LRIP  as  an  extension  of  the  Engineering, 
Manufacturing  and  Development  (EMD)  Phase  usually  runs  a  minimum  of 
two  years. 

b.  The  PLS  has,  in  fact,  had  several  PRRs  at  the 
contractor's  facility  since  the  production  contract  award  to 
monitor  process. 

5.  Page  24,  Impact  of  Present  LRIP  Process. 

No  mention  was  made  of  the  fact  that  the  PLS  LRIP  quantities  are 
part  of  the  production  contract  and  that  the  contract  is  firm  fixed 
price,  with  the  contractor  absorbing  all  costs  associated  with  test 
deficiencies.  This  allocation  of  risk  placed  on  the  contractor, 
was  critical  in  the  LRIP  approval  decision.  Since  this  fact  is 
lacking  in  the  report,  the  figures  and  statements  made  on  page  32 
regarding  PLS  LRIP  quantities  is  highly  misleading  because  it 
implies  that  costs  and  risks  for  corrective  actions  on  the  trucks 
fall  on  the  Army. 

6.  Page  $3  t  54,  Appendix  E. 

Hot  certain  what  is  meant  by  Acquisition  Strategy  -  Hot  Adhered  To 
end  also  Controls  Hot  Followed,  under  Internal  Controls,  since  no 
reference  ie  cited. 

7.  Page  56,  Appendix  G. 

The  PLS  line  is  in  error.  The  PLS  LRIP  is  part  of  a  5  year  multi¬ 
year  contract  with  encompassing  full-rate  production.  The  LRIP 
portion  as  well  as  annual  awards  are  not  separate  contracts. 


74 


Department  of  the  Army  Comments 


Final  Report 
Reference 


5ARD-ZCS 

SUBJECT:  DoD  Inspector  General  Draft  Report,  Low-Rate  Initial 
Production  in  Major  Defense  Acquisition  Programs  (Project  No.  2AE- 
0026) 


t.  Page  57,  Appendix  H. 

The  program  structure  is  misrepresented.  There  was  no  requirement 
to  specify  LRIP  quantities  at  Milestone  II,  which  was  held  in  Mar 
1988,  LRIP  requirement  came  into  effect  in  the  FY90  Defense 
Authorisation  Act. 

9.  POC  for  this  action  is  Mr.  Steve  Rann,  SFAE-CS-L/SARD-ZCS, 
695-8442. 


'  Roy  D.  Lewis 
Colonel,  GS 
Director  for  Combat 
Service  Support 
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MEMORANDUM  FOR  ASSISTANT  INSPECTOR  GENERAL  FOR  AUDITING 
OFFICE  OF  THE  INSPECTOR  GENERAL 
DEPARTMENT  OF  DEFENSE 

SUBJECT.  Draft  Audit  Report  on  the  Air  Force  Milstar  Terminal  Program 
(PROJECT  NO.  2AE-0026) 

We  have  reviewed  the  subject  report  which  alleges  several  major  defense 
acquisition  programs  entered  Low  Rate  Initial  Production  (LRIP)  without  completing 
prerequisites  in  design,  testing,  and  preparation  for  production  (Finding  A).  It  also 
asserts  that  LRIP  acquisition  strategies  for  major  defense  acquisition  programs  do 
not  effectively  limit  production  quantities  and  control  program  risk  before  Milestone 
III,  Production  Approval  (Finding  B).  We  have  reviewed  the  facts  on  the  Milstar 
Terminal  Program  and  do  not  concur  with  the  information  presented  in  the  report  nor 
with  its  conclusions. 

The  information  in  this  draft  report  is  based  on  the  audit  results  documented  in 
Report  No.  93-084,  “Air  Force  Strategic  and  Tactical  Relay  Satellite  Terminal 
Program."  In  our  earlier  letter,  dated  23  March  1993,  we  provided  a  detailed  response 
to  the  issues  documented  in  Report  No.  93-084  and  are  still  waiting  for  the  resolution. 
It  is  premature  to  generate  a  new  report  that  is  based,  in  part,  on  information  that  is 
still  being  contested,  and  to  cite  examples  that  may  not  be  valid.  We  request  you  wait 
for  the  resolution  before  completing  the  new  report.  The  attached  detailed  comments 
reiterate  the  USAF  position,  but  are  arranged  to  match  the  organization  of  the  new 
report 

The  detailed  comments  explain  that  the  program  met  the  criteria  in  the  Program 
Management  Directive  that  were  required  to  demonstrate  readiness  for  LRIP.  This 
criterion  included  specific  measures  for  verifying  terminal  performance  as  well  as 
compatibility  with  several  space  systems  and  other  service  terminals.  The  results 
were  reported  to  the  Defense  Acquisition  Board  (DAB)  in  a  1989  Milestone  II1A 
review.  The  DAB  concurred  that  the  program  was  ready  to  enter  into  LRIP.  Facts 
continue  to  support  this  decision.  LRIP  functioned  as  intended  by  resolving 
manufacturing  issues;  providing  terminals  for  Initial  Operation  Test  and  Evaluation; 
and  forming  the  competitive  basis  for  a  contract  award  that  lowered  terminal  unit 
prices  by  one  half.  Additionally,  there  have  been  very  few  Engineering  Change 
Proposals  generated  6ince  the  award  of  the  LRIP  contracts  demonstrating  the 
stability  of  the  design. 

The  detailed  comments  also  clearly  establish  why  completing  the  terminal 
procurement  under  the  LRIP  contract  was  in  the  Government's  best  interest  and  did 
not  induce  excessive  risk  to  the  program.  This  was  a  major  issue  for  the  October 
1992  Milstar  Restructure  DAB.  After  considerable  review  of  the  issue,  the  Director 
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of  Operational  Test  and  Evaluation  and  Assistant  Secretary  of  Defense  for 
Production  and  Logistics  staffs,  concluded  the  risk  of  completing  production  under  the 
LREP  contract  did  not  warrant  the  added  expense  associated  with  an  18  month  delay. 
This  recommendation  was  conditioned  on  successful  completion  of  an  independent 
Operational  Assessment  (OA)  and  Reliability-Qualification  (REL-QUAL)  test.  Given 
the  completion  of  the  OA,  which  found  no  show  stoppers,  and  the  successful 
completion  of  the  CPT  REL-QUAL  test,  the  buy-out  contract  was  awarded. 

We  look  forward  to  resolving  your  concerns  and  looking  for  ways  to  improve  the 
acquisition  process.  If  there  are  any  further  questions,  your  staff  should  contact 
Capt  Cox,  SAF/AQSS,  at  extension  33364  or  Lt  Col  Wain,  AFPEO/SP,  at  extension 
38056. 


ATCH 

Detailed  Comments 


DARLEEN  A.  DRUYUN 
Deputy  Assistant  Secretary 
(Acquisition) 


cc:  USD(A) 

ASD  (C3I) 
DOT&E 

COMPTROLLER 

AF/TE 

AFPEO/SP 

AFOTEC/CV 

SMC/MC 

ESC/MS 

JTPO 
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Final  Report 

Reference  ■■■■■  —  - - - 

Detailed  Response  to  Finding  A 
"Readiness  for  Low-Rate  Initial  Production" 

Major  Defense  acquisition  programs  enured  UUP  without  completing  prerequisites 
in  design,  testing  and  preparation  for  production ,  Premature  entry  into  UUP  was 
caused  by  inadequacies  in  the  milestone  review  process,  regulations ,  and  policy 
guidance  for  UUP.  Also ,  program  planning  was  done  when  urgency  to  meet 
threats  justified  highly  concurrent  development  and  production  efforts .  As  a  result, 
the  Government  incurred  significant  program  risk  from  systems  entering  UUP 
when  their  designs  were  not  stable  and  the  readiness  of  production  processes  was 
not  verified 

The  Milstar  Program  Management  Directive  included  specific  criteria  for  verifying  terminal 
performance,  compatibility  with  several  space  systems  and  other  service  terminals,  and  readiness 
for  production.  These  criteria  were  met,  and  results  of  development  progress  to  date  were  reported 
to  the  DAB  at  a  Milestone  IDA  review  in  May  1989.  The  DAB  determined  that  the  design  reviews, 
developmental  and  early  operational  testing,  and  production  readiness  reviews  had  been 
successfully  accomplished  as  required  As  a  result,  the  DAB  authorized  the  Air  Force  to  proceed 
with  LRP  for  Milstar  Command  Post  Terminals. 


Planning  and  Preparation  for  Low-Rate  Initial  Production 

1 1  Design  Problems  (set  page  12) 

The  Department  of  Defense  (DoD)  Inspector  GeneraVs  (JG's)  report  states  that  the  Air  Force 
Milstar  Terminal  Program  entered  UUP  with  an  incomplete  design.  As  evidence,  the  report  cites 
the  foci  that  UUP  contracts  were  awarded  with  B  level  specifications  ( performance  requirements) 
rather  than  C  level  specifications  (design  drawings). 

As  determined  through  the  DAB  review  process,  there  were  no  unresolved  design  issues  in 
May  1989  that  would  have  prevented  the  start  of  LRIP.  All  hardware  FCA/PCAs  had  been 
accomplished,  with  closure  plans  in  place  for  all  action  items.  Design  and  assembly  drawings 
were  available  to  the  LRIP  contractors  and  would  be  finalized  upon  closure  of  PCA  action  items. 
Three  of  four  software  configuration  items  were  complete;  testing  of  the  fourth  item  was 
underway  B  level  specifications  were  used  to  award  the  LRIP  contracts  because  the  C  level 
specifications  were  not  finalized  until  after  reliability  qualification  testing  had  been  completed;  as 
anticipated,  this  resulted  in  some  drawing  updates  due  to  minor  design  corrections  that  were  easily 
incorporated  into  the  production  baseline.  The  terminal's  detailed  design  was  quite  stable  at  the 
time  of  LRIP  award. 

-  Additionally  the  draft  IG  report  states  the  contractor  was  still  completing  design  as  of 

November  1992,  as  evidenced  by  open  product  quality  deficiencies,  which  could  * result  in 
hazardous  conditions  for  users  or  prevent  performance  of  strategic  or  tactical  functions.  ” 

Product  Quality  Deficiency  Reports  (PQDR)  are  not  an  accurate  measure  of  design  maturity. 
Only  one  of  the  PQDRs  initiated  in  over  18  months  of  user  testing  was  deemed  Category  I, 
affecting  the  ability  of  the  terminal  to  perform  specific  strategic  or  tactical  functions;  that  problem 
had  already  been  corrected  in  the  next  upgrade  of  the  terminal  software  at  the  time  of  detection 
Most  of  the  PQDRs  opened  to  date  represent  user  requested  enhancements  to  the  system  and  are 
not  deficiencies  against  the  terminal  system  specifications,  nor  do  they  represent  an  incomplete 
design.  Changes  of  this  type  are  expected  to  occur  continuously  as  the  user  gains  more  experience 
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Filial  Report 


Reference 


De I eted 
(See  31) 


19 


review  functional  performance  testing,  environmental  testing,  electromagnetic  interference  testing, 
and  TEMPEST  testing  had  been  successfully  completed.  Right  tests  with  an  wrbome  terminal 
aboard  the  (4950th  Test  Wing’s  C-18)  had  been  successfully  underway  since  September  1988. 
Survivabiliry/vulnerability  and  reliability  jrowth  testing  was  underway  with  no  major  design  issues 
expected  to  be  found  (none  were).  Operational  (terminal  control)  software  had  received 
government  led  verification  and  validation,  with  extensive  user  involvement.  In  summary, 
adequate  test  data  was  available  to  support  a  DAB  go-ahead. 


Review  Performance  (see  page  19) 

The  IG  report  alleges  that  the  Air  Force  Milstar  Terminals  Program  Office  did  not  perform 
annual  incremental  Production  Readiness  Reviews  during  engineering  and  manufacturing 
development  as  required  by  MJL-STD-1521B . 

The  IG’s  assessment  of  the  program  and  interpretation  of  the  Military  Standard  is  incorrect  for 
the  following  reasons 

a)  The  program  office  had  conducted  two  Production  Readiness  Reviews  (PRRs).  The  first 
PRR  was  accomplished  during  March/April  1988;  the  second  PRR  was  conducted  March  1989. 
Both  reviews  were  prior  to  the  LRIP  DAB  review  in  May  1989.  Both  reviews  were  highly 
successful  and  showed  that  production  processes  were  ready  to  be  implemented.  Significant 
features  of  the  Milstar  terminal  program's  transition  to  production  included  use  of  low  risk 
hardware  and  software  technology,  use  of  standard  parts,  and  early  use  of  manufacturing  lines  to 
build  EDMs  As  reported  to  the  DAB  in  May  1989,  factory  test  sets  were  in  place,  tooling  was  on 
line,  manufacturing  processes  and  work  instructions  were  complete,  environmental  stress 
screening  requirements  were  established,  and  critical  piece  parts/vendors/hardness  items  were 
identified 

b)  Also,  prior  to  the  award  of  the  LRIP  contracts,  during  the  Full  Scale  Engineering 
Development  contract,  the  program  office  conducted  Manufacturing  Assessment  Reviews  (MARs) 
and  Leadcr/Follower  meetings  bimonthly.  The  MARs  provided  the  program  office  nearly 
continuous  visibility  into  the  contractor's  production  abilities.  The  leader/follower  meetings  were 
critical  to  assure  transfer  of  product  and  process  technology  between  the  two  potential  production 
contractors 

c)  Lastly,  current  LRIP  status  shows  both  Raytheon  and  Rockwell  are  achieving  the  contract 
schedule  and  are  anticipating  early  delivery  of  the  remaining  LRIP  quantity.  This  is  the  most 
credible  demonstration  that  production  readiness  had  been  achieved  prior  to  the  award  of  the  LRIP 
contracts.  Raytheon  is  projecting  a  S20M  cost  underrun  ,  while  Rockwell  is  anticipating  a  $10M 
underrun.  The  Government  can  mitigate  risk  to  insure  production  readiness  through  the  use  of 
PRRs,  MARs,  and  regulations  but  there  is  no  substitute  for  actual  delivery  of  hardware. 


EngiPtCling  Development  Models  (see  page  21) 

The  IG  report  states  that  Milstar  terminal  EDMs  available  for  testing  before  the  June  1989 UUP 
decision  were  not  fully  mature  in  design.  Contractor  schedule  documentation  relating  to  the  EMD 
contract  showed  that  deliveries  affinal  versions  of  the  development  models  did  not  begin  until 
1991. 

Forma)  performance  and  qualification  testing  was  completed,  before  the  LRIP  decision,  on 
hardware  that  included  all  design  fixes  identified  through  the  development  and  testing  process. 
Test  data  was  generated,  in  general,  from  the  Full  Scale  Engineering  Development  model  terminals 
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that  included  these  fixes  Over  4000  hours  of  terminal,  term inal  to-pay load,  and  terminaJ-to- 
terminai  testing  had  been  successfully  accomplished.  All  user  functionality  requirements  tod  been 
provided  and  the  terminal's  communications  performance  met  or  exceeded  specification 
requirements.  Box  level  reliability  testing  showed  reliability  many  nines  higher  than  AFSATCOM 


requirements 
terminals 


The  final  versions  of  the  EDMs  mentioned  in  the  IG  report  refer  to  deliveries  against  a  contract 
line  item  intended  to  incorporate  all  the  design  fixes  identified  through  testing  into  hardware  that 
would  be  fully  production  representative.  The  contractor  has  modifiedthese  terminals  so  that  they 
support  all  the  functionality  present  in  actual  production  systems.  These  final  version  EDM 
terminals  are  being  used  to  support  early  operational  testing  on  aircraft  and  pre-launch  intersegment 
testing  with  the  Milstar  satellite  and  control  elements.  They  will  also  be  used,  along  with  LRIP 
terminals,  to  support  IOT&E  following  the  first  Milstar  satellite's  launch. 


Entry  into  Low-Rate  Initial  Production 
Acquisition  Planning  (see  page  23) 

The  1C  report  criticizes  the  decision  to  grant  LRIP  authority  for  Milstar  terminals  because  the 
urgency  of  providing  jam- resistant,  nuclear -surviv able  communications  service  was  a  significant 
factor  in  the  decision ,  despite  the  terminal  design  being  incomplete. 

This  implication  is  false  Specific  Program  Management  Directive  (PMD)  criteria  (what  would 
now  be  called  exit  criteria)  had  been  established  to  ensure  development  had  progressed  to  die  point 
when  initial  production  could  sun  These  criteria  were  1)  design  verification  against  the  Milstar 
Air  Force  Terminal  System  Specification,  2)  compatibility  with  AFSATCOM  ana  Single  Channel 
Transponder  Payloads.  3)  compatibility  with  the  FLTSAT  EHF  Package,  4)  compatibility  with  the 
Milstar  EHF/UHF  payload,  5)  comprehensive  logistics  planning  for  organic  support,  and  6) 
coordination  of  AFSC/AFLOAFCC  planning  for  terminal  installations.  As  shown  above  (under 
"Design  Problems"),  the  design  of  the  terminal  was  not  incomplete  at  the  time  of  the  DAB  review 
in  May  1989.  Overwhelming  evidence  that  the  PMD  criteria  had  been  met  was  presented  by  the 
Program  Office  to  the  DAB.  The  DAB  determined  that  the  Program  Office  had  indeed  met  the 
PMD  criteria  to  allow  entry  into  the  LRIP  phase  of  the  program.  The  fact  that  deployment  of  LRIP 
terminals  would  satisfy  urgent  national  needs  for  jam -resistant,  survivable  communications  may 
have  been  a  factor  in  the  DAB  decision  and  as  such  is  an  appropriate  business  consideration  for  the 
DAB;  however,  the  DAB  still  determined  that  the  Milstar  terminal  program  had  met  critical 
developmental  milestones  as  well 


Impact  of  Present  LRIP  Process 

Us*  of  Fundi  (lee  p»*es  25-26) 

The  1C  report  states  that  the  Air  Force  did  not  provide  adequate  oversight  or  control  over  the 
expenditures  cfRDT&E  and  procurement  appropriations  on  the  Milstar  terminal  UUP  contracts  for 
the  terminals .  As  a  result  t  violations  cf  United  States  Code,  tide  31 1  section  1301  occurred ,  which 
requires  that  appropriations  be  applied  only  cf  thou  pwrpousfor  which  the  appropriations  were 
made. 

The  Air  Force  has  reviewed  this  issue,  as  requested  by  the  DoD  IG,  and  determined  that  there 
is  no  basis  for  believing  that  an  Antideficiency  Act  violation  occurred.  The  review  covered 
progress  payment  liquidation  against  deliveries  and  concluded  that  no  progress  payments  were 
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made  for  any  type  of  cost  in  excess  of  the  appropriations  available  to  pay  such  cost.  The  Contract 
Line  Item  Number  (CLIN)  structure  of  the  contract  contains  discrete  line  items  which  are 
associated  with  each  separate  type  of  appropriation.  Further,  the  contract  requires  that  incurred 
costs  be  segregated  by  appropriation.  This  is  accomplished  through  the  Contract  Funding  Status 
Report  (CFSR).  Funds  for  each  appropriation  have  been  obligated  to  cover  funding  liability  at  the 
ceiling  price  of  each  funded  CLIN.  Therefore,  there  are  sufficient  funds  available  for  payment  of 
the  contractor’s  incurred  costs  under  each  CLIN.  Accordingly,  the  Antideficiency  Act  was  not 
violated 

We  are  aware  of  the  resolution  to  a  disputed  recommendation  in  OIG  Audit  Report  No.  92- 
064,  “Titan  IV  Program,”  dated  March  23,  1992.  We  understand  that  in  accordance  with  that 
resolution,  OSD  plans  to  require  that  contractor’s  requests  for  payment  must  identify  amounts  by 
contract  line  items  or  subline  items  to  the  disbursing  officer.  When  such  new  contractor  guidelines 
art  required  by  governing  directives,  the  Air  Force  will  apply  them  to  the  Milstar  progress 
payments 
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The  OSD  staff  determined  that  the  buy-out  award  should  proceed  after  completion  of  reliability 
qualification  testing  and  an  independent  Operational  Assessment  (OA).  The  readiness  of  the 
terminal  for  this  final  production  was  explicitly  considered  by  the  OSD  staff  in  preparation  for  the 
DAB  The  Director  of  Operational  Test  and  Evaluation  (DOT&E)  staff  reviewed  the  program  in 
May  92  and  requested  AFOTEC  conduct  the  OA  of  the  CPT.  The  OA  was  required  to  '’address  all 
terminal  Test  and  Evaluation  Master  Plan  (TEMP)  OT&E  objectives  and  identify  any  'show- 
stoppers'  which  would  impact  successful  completion  of  the  AF  CPT  buy-out."  While  there  were 
several  issues  raised,  AFOTEC  concluded  there  were  no  "show  stoppers"  in  meeting  all  threshold 
requirements.  The  issues  raised  in  the  OA  have  been  addressed  and  were  considered  by  the  OSD 
staff  in  preparation  for  the  October  1992  DAB.  The  decision  of  AF/TE  and  the  OSD  stiff  was  the 
cost  of  a  delay  in  the  program  (S144M)  far  outweighed  the  small  risk  of  delaying  the  buy-out  until 
OT&E  could  be  accomplished  The  DAB  therefore  approved  the  award  of  contracts  to  procure  the 
remaining  terminals  Because  of  the  reduction  in  quantity,  the  remaining  38  terminals  could  not  be 
construed  as  a  full  rate  production  effort  and  certainly  could  not  justify  the  cost  of  a  break  in 
production  to  wait  for  a  satellite  IOT&E. 
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